{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MultiEval Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates a basic parameter sweep with LensKits `MultiEval` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "We first need to import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lenskit.batch import MultiEval\n",
    "from lenskit.crossfold import partition_users, SampleN\n",
    "from lenskit.algorithms import basic, als\n",
    "from lenskit.datasets import MovieLens\n",
    "from lenskit import topn, util\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Progress bars are useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\michaelekstrand\\Anaconda3\\envs\\lkpy-dev\\lib\\site-packages\\tqdm\\std.py:668: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  from pandas import Panel\n"
     ]
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a little while to run things, and can get kinda quiet in here. Let's set up logging so we can see the logging output in the notebook's message stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then set up the data access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlsmall = MovieLens('../data/ml-latest-small')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to run our evaluation and store its output in the `my-eval` directory, generating 20-item recommendation lists::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval = MultiEval('my-eval', recommend=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use a 5-fold cross-validation setup.  We save the data into a list in memory so we have access to the test data later.  In a larger experiment, you might write the partitions to disk and pass the file names to `add_datasets`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.crossfold partitioning 100004 rows for 671 users into 5 partitions\n",
      "[   INFO] lenskit.crossfold fold 0: selecting test ratings\n",
      "[   INFO] lenskit.crossfold fold 0: partitioning training data\n",
      "[   INFO] lenskit.crossfold fold 1: selecting test ratings\n",
      "[   INFO] lenskit.crossfold fold 1: partitioning training data\n",
      "[   INFO] lenskit.crossfold fold 2: selecting test ratings\n",
      "[   INFO] lenskit.crossfold fold 2: partitioning training data\n",
      "[   INFO] lenskit.crossfold fold 3: selecting test ratings\n",
      "[   INFO] lenskit.crossfold fold 3: partitioning training data\n",
      "[   INFO] lenskit.crossfold fold 4: selecting test ratings\n",
      "[   INFO] lenskit.crossfold fold 4: partitioning training data\n"
     ]
    }
   ],
   "source": [
    "pairs = list(partition_users(mlsmall.ratings, 5, SampleN(5)))\n",
    "eval.add_datasets(pairs, name='ML-Small')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to test explicit MF with several neighborhood sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval.add_algorithms([als.BiasedMF(f) for f in [20, 30, 40, 50]],\n",
    "                    attrs=['features'], name='BiasedMF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And implicit MF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval.add_algorithms([als.ImplicitMF(f) for f in [20, 30, 40, 50]],\n",
    "                    attrs=['features'], name='ImplicitMF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And add a popular baseline for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval.add_algorithms(basic.Popular(), name='Pop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we will run the experiment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "67de2deea99e4eabb55a5bea38cd94ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=45.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._multi starting run 1: als.BiasedMF(features=20, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=20, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=20, regularization=0.1) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99329 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9060 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 351ms] normalizing 671x9060 matrix (99329 nnz)\n",
      "[   INFO] lenskit.algorithms.als [1.75s] training biased MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als [5.15s] finished epoch 0 (|ΔP|=32.144, |ΔQ|=124.881)\n",
      "[   INFO] lenskit.algorithms.als [5.17s] finished epoch 1 (|ΔP|=12.154, |ΔQ|=51.625)\n",
      "[   INFO] lenskit.algorithms.als [5.18s] finished epoch 2 (|ΔP|=8.825, |ΔQ|=27.235)\n",
      "[   INFO] lenskit.algorithms.als [5.19s] finished epoch 3 (|ΔP|=6.144, |ΔQ|=17.365)\n",
      "[   INFO] lenskit.algorithms.als [5.20s] finished epoch 4 (|ΔP|=4.521, |ΔQ|=12.784)\n",
      "[   INFO] lenskit.algorithms.als [5.22s] finished epoch 5 (|ΔP|=3.526, |ΔQ|=9.939)\n",
      "[   INFO] lenskit.algorithms.als [5.24s] finished epoch 6 (|ΔP|=2.876, |ΔQ|=8.045)\n",
      "[   INFO] lenskit.algorithms.als [5.25s] finished epoch 7 (|ΔP|=2.418, |ΔQ|=6.724)\n",
      "[   INFO] lenskit.algorithms.als [5.27s] finished epoch 8 (|ΔP|=2.080, |ΔQ|=5.767)\n",
      "[   INFO] lenskit.algorithms.als [5.28s] finished epoch 9 (|ΔP|=1.823, |ΔQ|=5.051)\n",
      "[   INFO] lenskit.algorithms.als [5.29s] finished epoch 10 (|ΔP|=1.623, |ΔQ|=4.494)\n",
      "[   INFO] lenskit.algorithms.als [5.31s] finished epoch 11 (|ΔP|=1.462, |ΔQ|=4.043)\n",
      "[   INFO] lenskit.algorithms.als [5.32s] finished epoch 12 (|ΔP|=1.326, |ΔQ|=3.665)\n",
      "[   INFO] lenskit.algorithms.als [5.33s] finished epoch 13 (|ΔP|=1.208, |ΔQ|=3.344)\n",
      "[   INFO] lenskit.algorithms.als [5.35s] finished epoch 14 (|ΔP|=1.104, |ΔQ|=3.068)\n",
      "[   INFO] lenskit.algorithms.als [5.36s] finished epoch 15 (|ΔP|=1.013, |ΔQ|=2.828)\n",
      "[   INFO] lenskit.algorithms.als [5.38s] finished epoch 16 (|ΔP|=0.932, |ΔQ|=2.619)\n",
      "[   INFO] lenskit.algorithms.als [5.39s] finished epoch 17 (|ΔP|=0.861, |ΔQ|=2.435)\n",
      "[   INFO] lenskit.algorithms.als [5.40s] finished epoch 18 (|ΔP|=0.797, |ΔQ|=2.270)\n",
      "[   INFO] lenskit.algorithms.als [5.41s] finished epoch 19 (|ΔP|=0.740, |ΔQ|=2.122)\n",
      "[   INFO] lenskit.algorithms.als trained model in 5.41s (|P|=29.700828, |Q|=94.427435)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=20, regularization=0.1) in 5.43s\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-u7bm2cqz.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 2.07s\n",
      "[   INFO] lenskit.batch._multi run 1: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-7t4oer0n.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=20, regularization=0.1) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.93s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.14s\n",
      "[   INFO] lenskit.batch._multi run 1: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 1: als.BiasedMF(features=20, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 2: als.BiasedMF(features=30, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=30, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=30, regularization=0.1) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99329 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9060 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 47ms] normalizing 671x9060 matrix (99329 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 57ms] training biased MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als [ 75ms] finished epoch 0 (|ΔP|=33.389, |ΔQ|=126.440)\n",
      "[   INFO] lenskit.algorithms.als [ 92ms] finished epoch 1 (|ΔP|=12.618, |ΔQ|=53.305)\n",
      "[   INFO] lenskit.algorithms.als [ 109ms] finished epoch 2 (|ΔP|=9.101, |ΔQ|=27.239)\n",
      "[   INFO] lenskit.algorithms.als [ 127ms] finished epoch 3 (|ΔP|=6.112, |ΔQ|=17.111)\n",
      "[   INFO] lenskit.algorithms.als [ 144ms] finished epoch 4 (|ΔP|=4.402, |ΔQ|=12.463)\n",
      "[   INFO] lenskit.algorithms.als [ 161ms] finished epoch 5 (|ΔP|=3.364, |ΔQ|=9.626)\n",
      "[   INFO] lenskit.algorithms.als [ 178ms] finished epoch 6 (|ΔP|=2.694, |ΔQ|=7.761)\n",
      "[   INFO] lenskit.algorithms.als [ 196ms] finished epoch 7 (|ΔP|=2.235, |ΔQ|=6.455)\n",
      "[   INFO] lenskit.algorithms.als [ 213ms] finished epoch 8 (|ΔP|=1.903, |ΔQ|=5.496)\n",
      "[   INFO] lenskit.algorithms.als [ 231ms] finished epoch 9 (|ΔP|=1.656, |ΔQ|=4.769)\n",
      "[   INFO] lenskit.algorithms.als [ 246ms] finished epoch 10 (|ΔP|=1.466, |ΔQ|=4.205)\n",
      "[   INFO] lenskit.algorithms.als [ 264ms] finished epoch 11 (|ΔP|=1.315, |ΔQ|=3.757)\n",
      "[   INFO] lenskit.algorithms.als [ 282ms] finished epoch 12 (|ΔP|=1.193, |ΔQ|=3.392)\n",
      "[   INFO] lenskit.algorithms.als [ 303ms] finished epoch 13 (|ΔP|=1.091, |ΔQ|=3.088)\n",
      "[   INFO] lenskit.algorithms.als [ 328ms] finished epoch 14 (|ΔP|=1.002, |ΔQ|=2.828)\n",
      "[   INFO] lenskit.algorithms.als [ 348ms] finished epoch 15 (|ΔP|=0.924, |ΔQ|=2.605)\n",
      "[   INFO] lenskit.algorithms.als [ 368ms] finished epoch 16 (|ΔP|=0.855, |ΔQ|=2.409)\n",
      "[   INFO] lenskit.algorithms.als [ 385ms] finished epoch 17 (|ΔP|=0.794, |ΔQ|=2.235)\n",
      "[   INFO] lenskit.algorithms.als [ 408ms] finished epoch 18 (|ΔP|=0.739, |ΔQ|=2.080)\n",
      "[   INFO] lenskit.algorithms.als [ 426ms] finished epoch 19 (|ΔP|=0.690, |ΔQ|=1.941)\n",
      "[   INFO] lenskit.algorithms.als trained model in  428ms (|P|=30.758338, |Q|=94.572805)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=30, regularization=0.1) in  444ms\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-2ndjw7ry.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 2.01s\n",
      "[   INFO] lenskit.batch._multi run 2: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-z40nh6m_.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=30, regularization=0.1) for 135 users (n_jobs=None)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.94s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.14s\n",
      "[   INFO] lenskit.batch._multi run 2: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 2: als.BiasedMF(features=30, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 3: als.BiasedMF(features=40, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=40, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=40, regularization=0.1) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99329 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9060 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 43ms] normalizing 671x9060 matrix (99329 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 53ms] training biased MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als [ 76ms] finished epoch 0 (|ΔP|=33.959, |ΔQ|=127.846)\n",
      "[   INFO] lenskit.algorithms.als [ 94ms] finished epoch 1 (|ΔP|=12.694, |ΔQ|=53.672)\n",
      "[   INFO] lenskit.algorithms.als [ 116ms] finished epoch 2 (|ΔP|=9.463, |ΔQ|=27.371)\n",
      "[   INFO] lenskit.algorithms.als [ 136ms] finished epoch 3 (|ΔP|=6.247, |ΔQ|=17.225)\n",
      "[   INFO] lenskit.algorithms.als [ 157ms] finished epoch 4 (|ΔP|=4.450, |ΔQ|=12.667)\n",
      "[   INFO] lenskit.algorithms.als [ 178ms] finished epoch 5 (|ΔP|=3.398, |ΔQ|=9.835)\n",
      "[   INFO] lenskit.algorithms.als [ 199ms] finished epoch 6 (|ΔP|=2.718, |ΔQ|=7.898)\n",
      "[   INFO] lenskit.algorithms.als [ 219ms] finished epoch 7 (|ΔP|=2.240, |ΔQ|=6.497)\n",
      "[   INFO] lenskit.algorithms.als [ 242ms] finished epoch 8 (|ΔP|=1.881, |ΔQ|=5.444)\n",
      "[   INFO] lenskit.algorithms.als [ 263ms] finished epoch 9 (|ΔP|=1.602, |ΔQ|=4.623)\n",
      "[   INFO] lenskit.algorithms.als [ 283ms] finished epoch 10 (|ΔP|=1.382, |ΔQ|=3.969)\n",
      "[   INFO] lenskit.algorithms.als [ 303ms] finished epoch 11 (|ΔP|=1.207, |ΔQ|=3.449)\n",
      "[   INFO] lenskit.algorithms.als [ 325ms] finished epoch 12 (|ΔP|=1.066, |ΔQ|=3.034)\n",
      "[   INFO] lenskit.algorithms.als [ 345ms] finished epoch 13 (|ΔP|=0.952, |ΔQ|=2.696)\n",
      "[   INFO] lenskit.algorithms.als [ 367ms] finished epoch 14 (|ΔP|=0.857, |ΔQ|=2.416)\n",
      "[   INFO] lenskit.algorithms.als [ 389ms] finished epoch 15 (|ΔP|=0.778, |ΔQ|=2.184)\n",
      "[   INFO] lenskit.algorithms.als [ 411ms] finished epoch 16 (|ΔP|=0.710, |ΔQ|=1.988)\n",
      "[   INFO] lenskit.algorithms.als [ 437ms] finished epoch 17 (|ΔP|=0.653, |ΔQ|=1.823)\n",
      "[   INFO] lenskit.algorithms.als [ 459ms] finished epoch 18 (|ΔP|=0.603, |ΔQ|=1.680)\n",
      "[   INFO] lenskit.algorithms.als [ 480ms] finished epoch 19 (|ΔP|=0.559, |ΔQ|=1.557)\n",
      "[   INFO] lenskit.algorithms.als trained model in  482ms (|P|=31.269929, |Q|=94.607805)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=40, regularization=0.1) in  497ms\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-twz757ey.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.94s\n",
      "[   INFO] lenskit.batch._multi run 3: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-h9t1159u.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=40, regularization=0.1) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.90s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.10s\n",
      "[   INFO] lenskit.batch._multi run 3: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 3: als.BiasedMF(features=40, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 4: als.BiasedMF(features=50, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=50, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=50, regularization=0.1) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99329 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9060 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 39ms] normalizing 671x9060 matrix (99329 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 51ms] training biased MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als [ 77ms] finished epoch 0 (|ΔP|=34.091, |ΔQ|=129.715)\n",
      "[   INFO] lenskit.algorithms.als [ 104ms] finished epoch 1 (|ΔP|=13.058, |ΔQ|=54.441)\n",
      "[   INFO] lenskit.algorithms.als [ 131ms] finished epoch 2 (|ΔP|=9.363, |ΔQ|=26.598)\n",
      "[   INFO] lenskit.algorithms.als [ 160ms] finished epoch 3 (|ΔP|=6.077, |ΔQ|=16.994)\n",
      "[   INFO] lenskit.algorithms.als [ 189ms] finished epoch 4 (|ΔP|=4.277, |ΔQ|=12.344)\n",
      "[   INFO] lenskit.algorithms.als [ 219ms] finished epoch 5 (|ΔP|=3.217, |ΔQ|=9.458)\n",
      "[   INFO] lenskit.algorithms.als [ 248ms] finished epoch 6 (|ΔP|=2.537, |ΔQ|=7.531)\n",
      "[   INFO] lenskit.algorithms.als [ 278ms] finished epoch 7 (|ΔP|=2.065, |ΔQ|=6.154)\n",
      "[   INFO] lenskit.algorithms.als [ 308ms] finished epoch 8 (|ΔP|=1.718, |ΔQ|=5.128)\n",
      "[   INFO] lenskit.algorithms.als [ 338ms] finished epoch 9 (|ΔP|=1.454, |ΔQ|=4.345)\n",
      "[   INFO] lenskit.algorithms.als [ 364ms] finished epoch 10 (|ΔP|=1.249, |ΔQ|=3.735)\n",
      "[   INFO] lenskit.algorithms.als [ 394ms] finished epoch 11 (|ΔP|=1.087, |ΔQ|=3.247)\n",
      "[   INFO] lenskit.algorithms.als [ 422ms] finished epoch 12 (|ΔP|=0.956, |ΔQ|=2.850)\n",
      "[   INFO] lenskit.algorithms.als [ 450ms] finished epoch 13 (|ΔP|=0.848, |ΔQ|=2.523)\n",
      "[   INFO] lenskit.algorithms.als [ 473ms] finished epoch 14 (|ΔP|=0.760, |ΔQ|=2.253)\n",
      "[   INFO] lenskit.algorithms.als [ 502ms] finished epoch 15 (|ΔP|=0.685, |ΔQ|=2.026)\n",
      "[   INFO] lenskit.algorithms.als [ 530ms] finished epoch 16 (|ΔP|=0.622, |ΔQ|=1.834)\n",
      "[   INFO] lenskit.algorithms.als [ 558ms] finished epoch 17 (|ΔP|=0.569, |ΔQ|=1.668)\n",
      "[   INFO] lenskit.algorithms.als [ 586ms] finished epoch 18 (|ΔP|=0.522, |ΔQ|=1.525)\n",
      "[   INFO] lenskit.algorithms.als [ 616ms] finished epoch 19 (|ΔP|=0.482, |ΔQ|=1.401)\n",
      "[   INFO] lenskit.algorithms.als trained model in  617ms (|P|=31.497696, |Q|=94.629892)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=50, regularization=0.1) in  633ms\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-3z9t_6nn.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 4: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-2cdh2vca.bpk\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=50, regularization=0.1) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.87s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.06s\n",
      "[   INFO] lenskit.batch._multi run 4: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 4: als.BiasedMF(features=50, regularization=0.1) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 5: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=20, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=20, reg=0.1, w=40) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 12ms] training implicit MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als have 99329 observations for 671 users and 9060 items\n",
      "[   INFO] lenskit.algorithms.als [7.19s] finished epoch 0 (|ΔP|=3394.400, |ΔQ|=8.617)\n",
      "[   INFO] lenskit.algorithms.als [7.24s] finished epoch 1 (|ΔP|=182.264, |ΔQ|=3.453)\n",
      "[   INFO] lenskit.algorithms.als [7.28s] finished epoch 2 (|ΔP|=72.796, |ΔQ|=2.408)\n",
      "[   INFO] lenskit.algorithms.als [7.33s] finished epoch 3 (|ΔP|=67.947, |ΔQ|=1.849)\n",
      "[   INFO] lenskit.algorithms.als [7.37s] finished epoch 4 (|ΔP|=79.469, |ΔQ|=1.574)\n",
      "[   INFO] lenskit.algorithms.als [7.42s] finished epoch 5 (|ΔP|=95.785, |ΔQ|=1.373)\n",
      "[   INFO] lenskit.algorithms.als [7.46s] finished epoch 6 (|ΔP|=78.213, |ΔQ|=1.164)\n",
      "[   INFO] lenskit.algorithms.als [7.51s] finished epoch 7 (|ΔP|=80.585, |ΔQ|=1.003)\n",
      "[   INFO] lenskit.algorithms.als [7.55s] finished epoch 8 (|ΔP|=76.428, |ΔQ|=0.873)\n",
      "[   INFO] lenskit.algorithms.als [7.60s] finished epoch 9 (|ΔP|=75.034, |ΔQ|=0.791)\n",
      "[   INFO] lenskit.algorithms.als [7.65s] finished epoch 10 (|ΔP|=65.064, |ΔQ|=0.711)\n",
      "[   INFO] lenskit.algorithms.als [7.70s] finished epoch 11 (|ΔP|=85.484, |ΔQ|=0.695)\n",
      "[   INFO] lenskit.algorithms.als [7.74s] finished epoch 12 (|ΔP|=60.615, |ΔQ|=0.600)\n",
      "[   INFO] lenskit.algorithms.als [7.79s] finished epoch 13 (|ΔP|=63.237, |ΔQ|=0.581)\n",
      "[   INFO] lenskit.algorithms.als [7.83s] finished epoch 14 (|ΔP|=56.944, |ΔQ|=0.538)\n",
      "[   INFO] lenskit.algorithms.als [7.88s] finished epoch 15 (|ΔP|=65.895, |ΔQ|=0.529)\n",
      "[   INFO] lenskit.algorithms.als [7.92s] finished epoch 16 (|ΔP|=54.112, |ΔQ|=0.495)\n",
      "[   INFO] lenskit.algorithms.als [7.97s] finished epoch 17 (|ΔP|=55.742, |ΔQ|=0.471)\n",
      "[   INFO] lenskit.algorithms.als [8.01s] finished epoch 18 (|ΔP|=59.130, |ΔQ|=0.455)\n",
      "[   INFO] lenskit.algorithms.als [8.06s] finished epoch 19 (|ΔP|=64.630, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [8.06s] finished training model with 20 features (|P|=2561.324204, |Q|=9.382931)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=20, reg=0.1, w=40) in 8.07s\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-dxt69lgp.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 2.43s\n",
      "[   INFO] lenskit.batch._multi run 5: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-t2nimnyr.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=20, reg=0.1, w=40) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.90s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.10s\n",
      "[   INFO] lenskit.batch._multi run 5: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 5: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 6: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=30, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=30, reg=0.1, w=40) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 15ms] training implicit MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als have 99329 observations for 671 users and 9060 items\n",
      "[   INFO] lenskit.algorithms.als [ 66ms] finished epoch 0 (|ΔP|=3823.401, |ΔQ|=8.176)\n",
      "[   INFO] lenskit.algorithms.als [ 116ms] finished epoch 1 (|ΔP|=197.089, |ΔQ|=3.204)\n",
      "[   INFO] lenskit.algorithms.als [ 166ms] finished epoch 2 (|ΔP|=112.848, |ΔQ|=2.102)\n",
      "[   INFO] lenskit.algorithms.als [ 216ms] finished epoch 3 (|ΔP|=120.734, |ΔQ|=1.643)\n",
      "[   INFO] lenskit.algorithms.als [ 267ms] finished epoch 4 (|ΔP|=128.267, |ΔQ|=1.291)\n",
      "[   INFO] lenskit.algorithms.als [ 317ms] finished epoch 5 (|ΔP|=123.323, |ΔQ|=1.038)\n",
      "[   INFO] lenskit.algorithms.als [ 368ms] finished epoch 6 (|ΔP|=103.669, |ΔQ|=0.860)\n",
      "[   INFO] lenskit.algorithms.als [ 418ms] finished epoch 7 (|ΔP|=97.214, |ΔQ|=0.746)\n",
      "[   INFO] lenskit.algorithms.als [ 468ms] finished epoch 8 (|ΔP|=92.921, |ΔQ|=0.677)\n",
      "[   INFO] lenskit.algorithms.als [ 519ms] finished epoch 9 (|ΔP|=85.104, |ΔQ|=0.591)\n",
      "[   INFO] lenskit.algorithms.als [ 571ms] finished epoch 10 (|ΔP|=84.155, |ΔQ|=0.555)\n",
      "[   INFO] lenskit.algorithms.als [ 624ms] finished epoch 11 (|ΔP|=84.841, |ΔQ|=0.519)\n",
      "[   INFO] lenskit.algorithms.als [ 674ms] finished epoch 12 (|ΔP|=80.127, |ΔQ|=0.482)\n",
      "[   INFO] lenskit.algorithms.als [ 723ms] finished epoch 13 (|ΔP|=75.243, |ΔQ|=0.449)\n",
      "[   INFO] lenskit.algorithms.als [ 774ms] finished epoch 14 (|ΔP|=79.453, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [ 824ms] finished epoch 15 (|ΔP|=72.474, |ΔQ|=0.406)\n",
      "[   INFO] lenskit.algorithms.als [ 874ms] finished epoch 16 (|ΔP|=70.143, |ΔQ|=0.389)\n",
      "[   INFO] lenskit.algorithms.als [ 924ms] finished epoch 17 (|ΔP|=64.952, |ΔQ|=0.362)\n",
      "[   INFO] lenskit.algorithms.als [ 974ms] finished epoch 18 (|ΔP|=86.699, |ΔQ|=0.384)\n",
      "[   INFO] lenskit.algorithms.als [1.02s] finished epoch 19 (|ΔP|=67.358, |ΔQ|=0.341)\n",
      "[   INFO] lenskit.algorithms.als [1.03s] finished training model with 30 features (|P|=2747.236676, |Q|=8.939173)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=30, reg=0.1, w=40) in 1.04s\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-08825cx6.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.97s\n",
      "[   INFO] lenskit.batch._multi run 6: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-smg_wepd.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=30, reg=0.1, w=40) for 135 users (n_jobs=None)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.09s\n",
      "[   INFO] lenskit.batch._multi run 6: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 6: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 7: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=40, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=40, reg=0.1, w=40) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 15ms] training implicit MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als have 99329 observations for 671 users and 9060 items\n",
      "[   INFO] lenskit.algorithms.als [ 67ms] finished epoch 0 (|ΔP|=4093.762, |ΔQ|=7.804)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 1 (|ΔP|=208.600, |ΔQ|=2.987)\n",
      "[   INFO] lenskit.algorithms.als [ 168ms] finished epoch 2 (|ΔP|=149.912, |ΔQ|=1.974)\n",
      "[   INFO] lenskit.algorithms.als [ 217ms] finished epoch 3 (|ΔP|=176.388, |ΔQ|=1.547)\n",
      "[   INFO] lenskit.algorithms.als [ 268ms] finished epoch 4 (|ΔP|=173.788, |ΔQ|=1.220)\n",
      "[   INFO] lenskit.algorithms.als [ 319ms] finished epoch 5 (|ΔP|=141.926, |ΔQ|=0.981)\n",
      "[   INFO] lenskit.algorithms.als [ 371ms] finished epoch 6 (|ΔP|=136.706, |ΔQ|=0.851)\n",
      "[   INFO] lenskit.algorithms.als [ 424ms] finished epoch 7 (|ΔP|=112.828, |ΔQ|=0.738)\n",
      "[   INFO] lenskit.algorithms.als [ 477ms] finished epoch 8 (|ΔP|=112.288, |ΔQ|=0.669)\n",
      "[   INFO] lenskit.algorithms.als [ 533ms] finished epoch 9 (|ΔP|=100.770, |ΔQ|=0.597)\n",
      "[   INFO] lenskit.algorithms.als [ 586ms] finished epoch 10 (|ΔP|=98.509, |ΔQ|=0.558)\n",
      "[   INFO] lenskit.algorithms.als [ 638ms] finished epoch 11 (|ΔP|=95.546, |ΔQ|=0.516)\n",
      "[   INFO] lenskit.algorithms.als [ 692ms] finished epoch 12 (|ΔP|=102.065, |ΔQ|=0.508)\n",
      "[   INFO] lenskit.algorithms.als [ 746ms] finished epoch 13 (|ΔP|=80.462, |ΔQ|=0.449)\n",
      "[   INFO] lenskit.algorithms.als [ 799ms] finished epoch 14 (|ΔP|=91.638, |ΔQ|=0.447)\n",
      "[   INFO] lenskit.algorithms.als [ 851ms] finished epoch 15 (|ΔP|=72.832, |ΔQ|=0.401)\n",
      "[   INFO] lenskit.algorithms.als [ 903ms] finished epoch 16 (|ΔP|=93.824, |ΔQ|=0.414)\n",
      "[   INFO] lenskit.algorithms.als [ 953ms] finished epoch 17 (|ΔP|=70.742, |ΔQ|=0.367)\n",
      "[   INFO] lenskit.algorithms.als [1.01s] finished epoch 18 (|ΔP|=84.424, |ΔQ|=0.379)\n",
      "[   INFO] lenskit.algorithms.als [1.06s] finished epoch 19 (|ΔP|=60.118, |ΔQ|=0.334)\n",
      "[   INFO] lenskit.algorithms.als [1.06s] finished training model with 40 features (|P|=2709.409153, |Q|=8.833095)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=40, reg=0.1, w=40) in 1.08s\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-w6e94233.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.88s\n",
      "[   INFO] lenskit.batch._multi run 7: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-28ty379_.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=40, reg=0.1, w=40) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.88s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.08s\n",
      "[   INFO] lenskit.batch._multi run 7: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 7: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 8: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=50, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=50, reg=0.1, w=40) on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 17ms] training implicit MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als have 99329 observations for 671 users and 9060 items\n",
      "[   INFO] lenskit.algorithms.als [ 70ms] finished epoch 0 (|ΔP|=4333.527, |ΔQ|=7.367)\n",
      "[   INFO] lenskit.algorithms.als [ 126ms] finished epoch 1 (|ΔP|=239.854, |ΔQ|=2.826)\n",
      "[   INFO] lenskit.algorithms.als [ 182ms] finished epoch 2 (|ΔP|=223.144, |ΔQ|=1.855)\n",
      "[   INFO] lenskit.algorithms.als [ 239ms] finished epoch 3 (|ΔP|=243.062, |ΔQ|=1.386)\n",
      "[   INFO] lenskit.algorithms.als [ 296ms] finished epoch 4 (|ΔP|=216.502, |ΔQ|=1.080)\n",
      "[   INFO] lenskit.algorithms.als [ 354ms] finished epoch 5 (|ΔP|=170.931, |ΔQ|=0.881)\n",
      "[   INFO] lenskit.algorithms.als [ 417ms] finished epoch 6 (|ΔP|=155.900, |ΔQ|=0.762)\n",
      "[   INFO] lenskit.algorithms.als [ 480ms] finished epoch 7 (|ΔP|=141.542, |ΔQ|=0.676)\n",
      "[   INFO] lenskit.algorithms.als [ 535ms] finished epoch 8 (|ΔP|=135.273, |ΔQ|=0.614)\n",
      "[   INFO] lenskit.algorithms.als [ 591ms] finished epoch 9 (|ΔP|=124.747, |ΔQ|=0.554)\n",
      "[   INFO] lenskit.algorithms.als [ 647ms] finished epoch 10 (|ΔP|=114.954, |ΔQ|=0.509)\n",
      "[   INFO] lenskit.algorithms.als [ 704ms] finished epoch 11 (|ΔP|=116.077, |ΔQ|=0.481)\n",
      "[   INFO] lenskit.algorithms.als [ 760ms] finished epoch 12 (|ΔP|=112.371, |ΔQ|=0.454)\n",
      "[   INFO] lenskit.algorithms.als [ 816ms] finished epoch 13 (|ΔP|=106.791, |ΔQ|=0.433)\n",
      "[   INFO] lenskit.algorithms.als [ 873ms] finished epoch 14 (|ΔP|=101.498, |ΔQ|=0.407)\n",
      "[   INFO] lenskit.algorithms.als [ 928ms] finished epoch 15 (|ΔP|=100.090, |ΔQ|=0.391)\n",
      "[   INFO] lenskit.algorithms.als [ 985ms] finished epoch 16 (|ΔP|=99.442, |ΔQ|=0.372)\n",
      "[   INFO] lenskit.algorithms.als [1.04s] finished epoch 17 (|ΔP|=91.748, |ΔQ|=0.361)\n",
      "[   INFO] lenskit.algorithms.als [1.10s] finished epoch 18 (|ΔP|=83.159, |ΔQ|=0.341)\n",
      "[   INFO] lenskit.algorithms.als [1.15s] finished epoch 19 (|ΔP|=91.485, |ΔQ|=0.346)\n",
      "[   INFO] lenskit.algorithms.als [1.16s] finished training model with 50 features (|P|=2585.461693, |Q|=8.413076)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=50, reg=0.1, w=40) in 1.17s\n",
      "[   INFO] lenskit.batch._multi generating 675 predictions for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-9v8s8_g5.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 675 predictions for 135 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 8: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-jlgqn3pq.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=50, reg=0.1, w=40) for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in 1.88s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.08s\n",
      "[   INFO] lenskit.batch._multi run 8: writing results to my-eval\\recommendations.parquet\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._multi finished run 8: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 9: Popular on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi training algorithm Popular on 99329 ratings\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99329 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm Popular in  13ms\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 135 users for Popular\n",
      "[   INFO] lenskit.sharing persisting Popular to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-u5seysx_.bpk\n",
      "[   INFO] binpickle.write pickled 967 bytes with 7 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with Popular for 135 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 135 users in  687ms\n",
      "[   INFO] lenskit.batch._multi generated recommendations in  757ms\n",
      "[   INFO] lenskit.batch._multi run 9: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 9: Popular on ML-Small:1\n",
      "[   INFO] lenskit.batch._multi starting run 10: als.BiasedMF(features=20, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=20, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=20, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.542\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 35ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 42ms] training biased MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als [ 51ms] finished epoch 0 (|ΔP|=32.182, |ΔQ|=123.970)\n",
      "[   INFO] lenskit.algorithms.als [ 62ms] finished epoch 1 (|ΔP|=12.051, |ΔQ|=53.112)\n",
      "[   INFO] lenskit.algorithms.als [ 75ms] finished epoch 2 (|ΔP|=8.789, |ΔQ|=27.769)\n",
      "[   INFO] lenskit.algorithms.als [ 88ms] finished epoch 3 (|ΔP|=6.172, |ΔQ|=17.540)\n",
      "[   INFO] lenskit.algorithms.als [ 100ms] finished epoch 4 (|ΔP|=4.598, |ΔQ|=12.869)\n",
      "[   INFO] lenskit.algorithms.als [ 112ms] finished epoch 5 (|ΔP|=3.569, |ΔQ|=9.991)\n",
      "[   INFO] lenskit.algorithms.als [ 126ms] finished epoch 6 (|ΔP|=2.877, |ΔQ|=8.118)\n",
      "[   INFO] lenskit.algorithms.als [ 139ms] finished epoch 7 (|ΔP|=2.393, |ΔQ|=6.813)\n",
      "[   INFO] lenskit.algorithms.als [ 152ms] finished epoch 8 (|ΔP|=2.042, |ΔQ|=5.843)\n",
      "[   INFO] lenskit.algorithms.als [ 166ms] finished epoch 9 (|ΔP|=1.779, |ΔQ|=5.092)\n",
      "[   INFO] lenskit.algorithms.als [ 179ms] finished epoch 10 (|ΔP|=1.574, |ΔQ|=4.490)\n",
      "[   INFO] lenskit.algorithms.als [ 192ms] finished epoch 11 (|ΔP|=1.408, |ΔQ|=3.997)\n",
      "[   INFO] lenskit.algorithms.als [ 205ms] finished epoch 12 (|ΔP|=1.272, |ΔQ|=3.588)\n",
      "[   INFO] lenskit.algorithms.als [ 219ms] finished epoch 13 (|ΔP|=1.157, |ΔQ|=3.247)\n",
      "[   INFO] lenskit.algorithms.als [ 232ms] finished epoch 14 (|ΔP|=1.060, |ΔQ|=2.959)\n",
      "[   INFO] lenskit.algorithms.als [ 244ms] finished epoch 15 (|ΔP|=0.977, |ΔQ|=2.713)\n",
      "[   INFO] lenskit.algorithms.als [ 257ms] finished epoch 16 (|ΔP|=0.903, |ΔQ|=2.499)\n",
      "[   INFO] lenskit.algorithms.als [ 270ms] finished epoch 17 (|ΔP|=0.838, |ΔQ|=2.310)\n",
      "[   INFO] lenskit.algorithms.als [ 284ms] finished epoch 18 (|ΔP|=0.779, |ΔQ|=2.141)\n",
      "[   INFO] lenskit.algorithms.als [ 297ms] finished epoch 19 (|ΔP|=0.726, |ΔQ|=1.991)\n",
      "[   INFO] lenskit.algorithms.als trained model in  298ms (|P|=29.679664, |Q|=94.403057)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=20, regularization=0.1) in  313ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-1ybb47q3.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.87s\n",
      "[   INFO] lenskit.batch._multi run 10: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-idr2npgt.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=20, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.08s\n",
      "[   INFO] lenskit.batch._multi run 10: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 10: als.BiasedMF(features=20, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 11: als.BiasedMF(features=30, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=30, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=30, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.542\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 38ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 47ms] training biased MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als [ 63ms] finished epoch 0 (|ΔP|=33.141, |ΔQ|=127.240)\n",
      "[   INFO] lenskit.algorithms.als [ 80ms] finished epoch 1 (|ΔP|=12.614, |ΔQ|=53.697)\n",
      "[   INFO] lenskit.algorithms.als [ 97ms] finished epoch 2 (|ΔP|=9.096, |ΔQ|=27.191)\n",
      "[   INFO] lenskit.algorithms.als [ 115ms] finished epoch 3 (|ΔP|=6.217, |ΔQ|=17.552)\n",
      "[   INFO] lenskit.algorithms.als [ 134ms] finished epoch 4 (|ΔP|=4.487, |ΔQ|=12.881)\n",
      "[   INFO] lenskit.algorithms.als [ 153ms] finished epoch 5 (|ΔP|=3.412, |ΔQ|=9.991)\n",
      "[   INFO] lenskit.algorithms.als [ 173ms] finished epoch 6 (|ΔP|=2.727, |ΔQ|=8.040)\n",
      "[   INFO] lenskit.algorithms.als [ 193ms] finished epoch 7 (|ΔP|=2.273, |ΔQ|=6.646)\n",
      "[   INFO] lenskit.algorithms.als [ 213ms] finished epoch 8 (|ΔP|=1.953, |ΔQ|=5.616)\n",
      "[   INFO] lenskit.algorithms.als [ 231ms] finished epoch 9 (|ΔP|=1.713, |ΔQ|=4.837)\n",
      "[   INFO] lenskit.algorithms.als [ 250ms] finished epoch 10 (|ΔP|=1.522, |ΔQ|=4.230)\n",
      "[   INFO] lenskit.algorithms.als [ 269ms] finished epoch 11 (|ΔP|=1.364, |ΔQ|=3.741)\n",
      "[   INFO] lenskit.algorithms.als [ 286ms] finished epoch 12 (|ΔP|=1.228, |ΔQ|=3.336)\n",
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 13 (|ΔP|=1.109, |ΔQ|=2.994)\n",
      "[   INFO] lenskit.algorithms.als [ 323ms] finished epoch 14 (|ΔP|=1.005, |ΔQ|=2.701)\n",
      "[   INFO] lenskit.algorithms.als [ 342ms] finished epoch 15 (|ΔP|=0.914, |ΔQ|=2.447)\n",
      "[   INFO] lenskit.algorithms.als [ 360ms] finished epoch 16 (|ΔP|=0.833, |ΔQ|=2.227)\n",
      "[   INFO] lenskit.algorithms.als [ 379ms] finished epoch 17 (|ΔP|=0.762, |ΔQ|=2.034)\n",
      "[   INFO] lenskit.algorithms.als [ 398ms] finished epoch 18 (|ΔP|=0.699, |ΔQ|=1.866)\n",
      "[   INFO] lenskit.algorithms.als [ 416ms] finished epoch 19 (|ΔP|=0.643, |ΔQ|=1.717)\n",
      "[   INFO] lenskit.algorithms.als trained model in  418ms (|P|=30.783977, |Q|=94.570775)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=30, regularization=0.1) in  432ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=30, regularization=0.1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-57mcycqu.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 11: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-zj2qmvbp.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=30, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.86s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.06s\n",
      "[   INFO] lenskit.batch._multi run 11: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 11: als.BiasedMF(features=30, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 12: als.BiasedMF(features=40, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=40, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=40, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.542\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 41ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 52ms] training biased MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als [ 74ms] finished epoch 0 (|ΔP|=33.550, |ΔQ|=128.060)\n",
      "[   INFO] lenskit.algorithms.als [ 93ms] finished epoch 1 (|ΔP|=12.835, |ΔQ|=54.282)\n",
      "[   INFO] lenskit.algorithms.als [ 116ms] finished epoch 2 (|ΔP|=9.471, |ΔQ|=27.924)\n",
      "[   INFO] lenskit.algorithms.als [ 140ms] finished epoch 3 (|ΔP|=6.239, |ΔQ|=17.265)\n",
      "[   INFO] lenskit.algorithms.als [ 161ms] finished epoch 4 (|ΔP|=4.385, |ΔQ|=12.487)\n",
      "[   INFO] lenskit.algorithms.als [ 185ms] finished epoch 5 (|ΔP|=3.309, |ΔQ|=9.559)\n",
      "[   INFO] lenskit.algorithms.als [ 208ms] finished epoch 6 (|ΔP|=2.624, |ΔQ|=7.616)\n",
      "[   INFO] lenskit.algorithms.als [ 231ms] finished epoch 7 (|ΔP|=2.152, |ΔQ|=6.267)\n",
      "[   INFO] lenskit.algorithms.als [ 254ms] finished epoch 8 (|ΔP|=1.811, |ΔQ|=5.294)\n",
      "[   INFO] lenskit.algorithms.als [ 278ms] finished epoch 9 (|ΔP|=1.556, |ΔQ|=4.565)\n",
      "[   INFO] lenskit.algorithms.als [ 302ms] finished epoch 10 (|ΔP|=1.360, |ΔQ|=3.997)\n",
      "[   INFO] lenskit.algorithms.als [ 325ms] finished epoch 11 (|ΔP|=1.206, |ΔQ|=3.539)\n",
      "[   INFO] lenskit.algorithms.als [ 349ms] finished epoch 12 (|ΔP|=1.081, |ΔQ|=3.159)\n",
      "[   INFO] lenskit.algorithms.als [ 373ms] finished epoch 13 (|ΔP|=0.978, |ΔQ|=2.838)\n",
      "[   INFO] lenskit.algorithms.als [ 396ms] finished epoch 14 (|ΔP|=0.891, |ΔQ|=2.565)\n",
      "[   INFO] lenskit.algorithms.als [ 419ms] finished epoch 15 (|ΔP|=0.818, |ΔQ|=2.331)\n",
      "[   INFO] lenskit.algorithms.als [ 440ms] finished epoch 16 (|ΔP|=0.754, |ΔQ|=2.129)\n",
      "[   INFO] lenskit.algorithms.als [ 463ms] finished epoch 17 (|ΔP|=0.699, |ΔQ|=1.954)\n",
      "[   INFO] lenskit.algorithms.als [ 487ms] finished epoch 18 (|ΔP|=0.651, |ΔQ|=1.803)\n",
      "[   INFO] lenskit.algorithms.als [ 509ms] finished epoch 19 (|ΔP|=0.608, |ΔQ|=1.671)\n",
      "[   INFO] lenskit.algorithms.als trained model in  512ms (|P|=31.243407, |Q|=94.662056)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=40, regularization=0.1) in  528ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-81n66289.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.87s\n",
      "[   INFO] lenskit.batch._multi run 12: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-3zf4d99v.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=40, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.95s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.15s\n",
      "[   INFO] lenskit.batch._multi run 12: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 12: als.BiasedMF(features=40, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 13: als.BiasedMF(features=50, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=50, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=50, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.542\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 40ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 52ms] training biased MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als [ 78ms] finished epoch 0 (|ΔP|=34.080, |ΔQ|=128.724)\n",
      "[   INFO] lenskit.algorithms.als [ 105ms] finished epoch 1 (|ΔP|=12.916, |ΔQ|=55.093)\n",
      "[   INFO] lenskit.algorithms.als [ 133ms] finished epoch 2 (|ΔP|=9.454, |ΔQ|=27.431)\n",
      "[   INFO] lenskit.algorithms.als [ 163ms] finished epoch 3 (|ΔP|=6.120, |ΔQ|=17.289)\n",
      "[   INFO] lenskit.algorithms.als [ 197ms] finished epoch 4 (|ΔP|=4.273, |ΔQ|=12.474)\n",
      "[   INFO] lenskit.algorithms.als [ 227ms] finished epoch 5 (|ΔP|=3.197, |ΔQ|=9.500)\n",
      "[   INFO] lenskit.algorithms.als [ 257ms] finished epoch 6 (|ΔP|=2.515, |ΔQ|=7.554)\n",
      "[   INFO] lenskit.algorithms.als [ 288ms] finished epoch 7 (|ΔP|=2.046, |ΔQ|=6.176)\n",
      "[   INFO] lenskit.algorithms.als [ 318ms] finished epoch 8 (|ΔP|=1.703, |ΔQ|=5.152)\n",
      "[   INFO] lenskit.algorithms.als [ 344ms] finished epoch 9 (|ΔP|=1.444, |ΔQ|=4.372)\n",
      "[   INFO] lenskit.algorithms.als [ 372ms] finished epoch 10 (|ΔP|=1.243, |ΔQ|=3.766)\n",
      "[   INFO] lenskit.algorithms.als [ 401ms] finished epoch 11 (|ΔP|=1.085, |ΔQ|=3.282)\n",
      "[   INFO] lenskit.algorithms.als [ 429ms] finished epoch 12 (|ΔP|=0.959, |ΔQ|=2.891)\n",
      "[   INFO] lenskit.algorithms.als [ 458ms] finished epoch 13 (|ΔP|=0.856, |ΔQ|=2.571)\n",
      "[   INFO] lenskit.algorithms.als [ 488ms] finished epoch 14 (|ΔP|=0.771, |ΔQ|=2.303)\n",
      "[   INFO] lenskit.algorithms.als [ 516ms] finished epoch 15 (|ΔP|=0.699, |ΔQ|=2.076)\n",
      "[   INFO] lenskit.algorithms.als [ 545ms] finished epoch 16 (|ΔP|=0.637, |ΔQ|=1.878)\n",
      "[   INFO] lenskit.algorithms.als [ 573ms] finished epoch 17 (|ΔP|=0.584, |ΔQ|=1.705)\n",
      "[   INFO] lenskit.algorithms.als [ 601ms] finished epoch 18 (|ΔP|=0.537, |ΔQ|=1.554)\n",
      "[   INFO] lenskit.algorithms.als [ 625ms] finished epoch 19 (|ΔP|=0.496, |ΔQ|=1.421)\n",
      "[   INFO] lenskit.algorithms.als trained model in  628ms (|P|=31.488355, |Q|=94.688959)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=50, regularization=0.1) in  643ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-y7cnds73.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.90s\n",
      "[   INFO] lenskit.batch._multi run 13: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-gd67chol.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=50, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.91s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.12s\n",
      "[   INFO] lenskit.batch._multi run 13: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 13: als.BiasedMF(features=50, regularization=0.1) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 14: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=20, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=20, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 12ms] training implicit MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 57ms] finished epoch 0 (|ΔP|=3402.517, |ΔQ|=8.849)\n",
      "[   INFO] lenskit.algorithms.als [ 105ms] finished epoch 1 (|ΔP|=165.373, |ΔQ|=3.576)\n",
      "[   INFO] lenskit.algorithms.als [ 177ms] finished epoch 2 (|ΔP|=70.845, |ΔQ|=2.407)\n",
      "[   INFO] lenskit.algorithms.als [ 228ms] finished epoch 3 (|ΔP|=62.977, |ΔQ|=1.824)\n",
      "[   INFO] lenskit.algorithms.als [ 277ms] finished epoch 4 (|ΔP|=68.344, |ΔQ|=1.582)\n",
      "[   INFO] lenskit.algorithms.als [ 325ms] finished epoch 5 (|ΔP|=74.716, |ΔQ|=1.407)\n",
      "[   INFO] lenskit.algorithms.als [ 376ms] finished epoch 6 (|ΔP|=74.762, |ΔQ|=1.202)\n",
      "[   INFO] lenskit.algorithms.als [ 426ms] finished epoch 7 (|ΔP|=71.862, |ΔQ|=1.026)\n",
      "[   INFO] lenskit.algorithms.als [ 473ms] finished epoch 8 (|ΔP|=69.738, |ΔQ|=0.895)\n",
      "[   INFO] lenskit.algorithms.als [ 521ms] finished epoch 9 (|ΔP|=63.080, |ΔQ|=0.803)\n",
      "[   INFO] lenskit.algorithms.als [ 569ms] finished epoch 10 (|ΔP|=63.476, |ΔQ|=0.725)\n",
      "[   INFO] lenskit.algorithms.als [ 616ms] finished epoch 11 (|ΔP|=61.933, |ΔQ|=0.663)\n",
      "[   INFO] lenskit.algorithms.als [ 663ms] finished epoch 12 (|ΔP|=56.981, |ΔQ|=0.601)\n",
      "[   INFO] lenskit.algorithms.als [ 710ms] finished epoch 13 (|ΔP|=58.126, |ΔQ|=0.581)\n",
      "[   INFO] lenskit.algorithms.als [ 757ms] finished epoch 14 (|ΔP|=55.184, |ΔQ|=0.543)\n",
      "[   INFO] lenskit.algorithms.als [ 803ms] finished epoch 15 (|ΔP|=53.067, |ΔQ|=0.506)\n",
      "[   INFO] lenskit.algorithms.als [ 850ms] finished epoch 16 (|ΔP|=46.104, |ΔQ|=0.461)\n",
      "[   INFO] lenskit.algorithms.als [ 897ms] finished epoch 17 (|ΔP|=53.481, |ΔQ|=0.470)\n",
      "[   INFO] lenskit.algorithms.als [ 943ms] finished epoch 18 (|ΔP|=43.898, |ΔQ|=0.417)\n",
      "[   INFO] lenskit.algorithms.als [ 991ms] finished epoch 19 (|ΔP|=46.695, |ΔQ|=0.420)\n",
      "[   INFO] lenskit.algorithms.als [ 992ms] finished training model with 20 features (|P|=2693.052181, |Q|=9.918696)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=20, reg=0.1, w=40) in 1.01s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-6g8mziiu.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 14: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-1mftac7d.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=20, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.92s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.12s\n",
      "[   INFO] lenskit.batch._multi run 14: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 14: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 15: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=30, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=30, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 14ms] training implicit MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 64ms] finished epoch 0 (|ΔP|=3839.187, |ΔQ|=8.261)\n",
      "[   INFO] lenskit.algorithms.als [ 115ms] finished epoch 1 (|ΔP|=196.361, |ΔQ|=3.208)\n",
      "[   INFO] lenskit.algorithms.als [ 168ms] finished epoch 2 (|ΔP|=105.465, |ΔQ|=2.146)\n",
      "[   INFO] lenskit.algorithms.als [ 219ms] finished epoch 3 (|ΔP|=111.411, |ΔQ|=1.681)\n",
      "[   INFO] lenskit.algorithms.als [ 272ms] finished epoch 4 (|ΔP|=125.129, |ΔQ|=1.373)\n",
      "[   INFO] lenskit.algorithms.als [ 323ms] finished epoch 5 (|ΔP|=119.156, |ΔQ|=1.098)\n",
      "[   INFO] lenskit.algorithms.als [ 376ms] finished epoch 6 (|ΔP|=106.082, |ΔQ|=0.924)\n",
      "[   INFO] lenskit.algorithms.als [ 430ms] finished epoch 7 (|ΔP|=94.524, |ΔQ|=0.806)\n",
      "[   INFO] lenskit.algorithms.als [ 486ms] finished epoch 8 (|ΔP|=93.629, |ΔQ|=0.724)\n",
      "[   INFO] lenskit.algorithms.als [ 540ms] finished epoch 9 (|ΔP|=88.710, |ΔQ|=0.644)\n",
      "[   INFO] lenskit.algorithms.als [ 595ms] finished epoch 10 (|ΔP|=83.786, |ΔQ|=0.609)\n",
      "[   INFO] lenskit.algorithms.als [ 648ms] finished epoch 11 (|ΔP|=84.301, |ΔQ|=0.561)\n",
      "[   INFO] lenskit.algorithms.als [ 700ms] finished epoch 12 (|ΔP|=82.129, |ΔQ|=0.531)\n",
      "[   INFO] lenskit.algorithms.als [ 752ms] finished epoch 13 (|ΔP|=75.127, |ΔQ|=0.490)\n",
      "[   INFO] lenskit.algorithms.als [ 803ms] finished epoch 14 (|ΔP|=78.422, |ΔQ|=0.483)\n",
      "[   INFO] lenskit.algorithms.als [ 855ms] finished epoch 15 (|ΔP|=67.931, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [ 909ms] finished epoch 16 (|ΔP|=77.571, |ΔQ|=0.448)\n",
      "[   INFO] lenskit.algorithms.als [ 959ms] finished epoch 17 (|ΔP|=65.900, |ΔQ|=0.408)\n",
      "[   INFO] lenskit.algorithms.als [1.01s] finished epoch 18 (|ΔP|=76.291, |ΔQ|=0.417)\n",
      "[   INFO] lenskit.algorithms.als [1.06s] finished epoch 19 (|ΔP|=54.487, |ΔQ|=0.364)\n",
      "[   INFO] lenskit.algorithms.als [1.07s] finished training model with 30 features (|P|=2769.199197, |Q|=9.173332)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=30, reg=0.1, w=40) in 1.08s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-wujsxbg4.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.91s\n",
      "[   INFO] lenskit.batch._multi run 15: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-g5rk95rj.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=30, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.97s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.16s\n",
      "[   INFO] lenskit.batch._multi run 15: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 15: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 16: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=40, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=40, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 16ms] training implicit MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 66ms] finished epoch 0 (|ΔP|=4128.532, |ΔQ|=7.637)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 1 (|ΔP|=220.165, |ΔQ|=2.979)\n",
      "[   INFO] lenskit.algorithms.als [ 170ms] finished epoch 2 (|ΔP|=168.248, |ΔQ|=1.970)\n",
      "[   INFO] lenskit.algorithms.als [ 225ms] finished epoch 3 (|ΔP|=194.807, |ΔQ|=1.533)\n",
      "[   INFO] lenskit.algorithms.als [ 280ms] finished epoch 4 (|ΔP|=171.658, |ΔQ|=1.149)\n",
      "[   INFO] lenskit.algorithms.als [ 334ms] finished epoch 5 (|ΔP|=156.670, |ΔQ|=0.946)\n",
      "[   INFO] lenskit.algorithms.als [ 388ms] finished epoch 6 (|ΔP|=131.727, |ΔQ|=0.797)\n",
      "[   INFO] lenskit.algorithms.als [ 442ms] finished epoch 7 (|ΔP|=123.673, |ΔQ|=0.698)\n",
      "[   INFO] lenskit.algorithms.als [ 496ms] finished epoch 8 (|ΔP|=111.367, |ΔQ|=0.620)\n",
      "[   INFO] lenskit.algorithms.als [ 551ms] finished epoch 9 (|ΔP|=108.997, |ΔQ|=0.569)\n",
      "[   INFO] lenskit.algorithms.als [ 602ms] finished epoch 10 (|ΔP|=102.912, |ΔQ|=0.519)\n",
      "[   INFO] lenskit.algorithms.als [ 655ms] finished epoch 11 (|ΔP|=102.266, |ΔQ|=0.493)\n",
      "[   INFO] lenskit.algorithms.als [ 707ms] finished epoch 12 (|ΔP|=93.307, |ΔQ|=0.450)\n",
      "[   INFO] lenskit.algorithms.als [ 759ms] finished epoch 13 (|ΔP|=100.050, |ΔQ|=0.446)\n",
      "[   INFO] lenskit.algorithms.als [ 813ms] finished epoch 14 (|ΔP|=86.456, |ΔQ|=0.404)\n",
      "[   INFO] lenskit.algorithms.als [ 865ms] finished epoch 15 (|ΔP|=96.727, |ΔQ|=0.403)\n",
      "[   INFO] lenskit.algorithms.als [ 918ms] finished epoch 16 (|ΔP|=80.487, |ΔQ|=0.365)\n",
      "[   INFO] lenskit.algorithms.als [ 972ms] finished epoch 17 (|ΔP|=93.062, |ΔQ|=0.371)\n",
      "[   INFO] lenskit.algorithms.als [1.03s] finished epoch 18 (|ΔP|=78.584, |ΔQ|=0.337)\n",
      "[   INFO] lenskit.algorithms.als [1.08s] finished epoch 19 (|ΔP|=95.092, |ΔQ|=0.348)\n",
      "[   INFO] lenskit.algorithms.als [1.08s] finished training model with 40 features (|P|=2662.006852, |Q|=8.555862)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=40, reg=0.1, w=40) in 1.09s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-yl2nyvs7.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 16: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-zof45ikk.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=40, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.88s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.08s\n",
      "[   INFO] lenskit.batch._multi run 16: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 16: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 17: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=50, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=50, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 16ms] training implicit MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 69ms] finished epoch 0 (|ΔP|=4303.915, |ΔQ|=7.684)\n",
      "[   INFO] lenskit.algorithms.als [ 126ms] finished epoch 1 (|ΔP|=223.686, |ΔQ|=2.929)\n",
      "[   INFO] lenskit.algorithms.als [ 185ms] finished epoch 2 (|ΔP|=191.189, |ΔQ|=1.885)\n",
      "[   INFO] lenskit.algorithms.als [ 248ms] finished epoch 3 (|ΔP|=221.345, |ΔQ|=1.397)\n",
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 4 (|ΔP|=195.121, |ΔQ|=1.084)\n",
      "[   INFO] lenskit.algorithms.als [ 361ms] finished epoch 5 (|ΔP|=172.094, |ΔQ|=0.887)\n",
      "[   INFO] lenskit.algorithms.als [ 418ms] finished epoch 6 (|ΔP|=146.067, |ΔQ|=0.761)\n",
      "[   INFO] lenskit.algorithms.als [ 474ms] finished epoch 7 (|ΔP|=137.360, |ΔQ|=0.672)\n",
      "[   INFO] lenskit.algorithms.als [ 529ms] finished epoch 8 (|ΔP|=133.786, |ΔQ|=0.603)\n",
      "[   INFO] lenskit.algorithms.als [ 585ms] finished epoch 9 (|ΔP|=113.976, |ΔQ|=0.538)\n",
      "[   INFO] lenskit.algorithms.als [ 641ms] finished epoch 10 (|ΔP|=113.613, |ΔQ|=0.501)\n",
      "[   INFO] lenskit.algorithms.als [ 697ms] finished epoch 11 (|ΔP|=107.685, |ΔQ|=0.463)\n",
      "[   INFO] lenskit.algorithms.als [ 753ms] finished epoch 12 (|ΔP|=98.233, |ΔQ|=0.431)\n",
      "[   INFO] lenskit.algorithms.als [ 810ms] finished epoch 13 (|ΔP|=106.997, |ΔQ|=0.429)\n",
      "[   INFO] lenskit.algorithms.als [ 873ms] finished epoch 14 (|ΔP|=92.050, |ΔQ|=0.392)\n",
      "[   INFO] lenskit.algorithms.als [ 931ms] finished epoch 15 (|ΔP|=105.957, |ΔQ|=0.386)\n",
      "[   INFO] lenskit.algorithms.als [ 987ms] finished epoch 16 (|ΔP|=90.454, |ΔQ|=0.357)\n",
      "[   INFO] lenskit.algorithms.als [1.04s] finished epoch 17 (|ΔP|=97.569, |ΔQ|=0.362)\n",
      "[   INFO] lenskit.algorithms.als [1.10s] finished epoch 18 (|ΔP|=75.906, |ΔQ|=0.325)\n",
      "[   INFO] lenskit.algorithms.als [1.16s] finished epoch 19 (|ΔP|=100.693, |ΔQ|=0.350)\n",
      "[   INFO] lenskit.algorithms.als [1.16s] finished training model with 50 features (|P|=2664.711758, |Q|=8.599825)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=50, reg=0.1, w=40) in 1.18s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-_p2cd7en.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 17: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-g5icm38l.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=50, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.93s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.14s\n",
      "[   INFO] lenskit.batch._multi run 17: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 17: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 18: Popular on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi training algorithm Popular on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm Popular in  14ms\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for Popular\n",
      "[   INFO] lenskit.sharing persisting Popular to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-1cf1hnwh.bpk\n",
      "[   INFO] binpickle.write pickled 967 bytes with 7 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with Popular for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in  739ms\n",
      "[   INFO] lenskit.batch._multi generated recommendations in  815ms\n",
      "[   INFO] lenskit.batch._multi run 18: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 18: Popular on ML-Small:2\n",
      "[   INFO] lenskit.batch._multi starting run 19: als.BiasedMF(features=20, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=20, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=20, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9056 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 44ms] normalizing 671x9056 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 54ms] training biased MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als [ 66ms] finished epoch 0 (|ΔP|=32.111, |ΔQ|=124.013)\n",
      "[   INFO] lenskit.algorithms.als [ 78ms] finished epoch 1 (|ΔP|=12.130, |ΔQ|=51.893)\n",
      "[   INFO] lenskit.algorithms.als [ 106ms] finished epoch 2 (|ΔP|=8.986, |ΔQ|=26.885)\n",
      "[   INFO] lenskit.algorithms.als [ 139ms] finished epoch 3 (|ΔP|=6.269, |ΔQ|=17.359)\n",
      "[   INFO] lenskit.algorithms.als [ 155ms] finished epoch 4 (|ΔP|=4.626, |ΔQ|=12.908)\n",
      "[   INFO] lenskit.algorithms.als [ 177ms] finished epoch 5 (|ΔP|=3.571, |ΔQ|=10.133)\n",
      "[   INFO] lenskit.algorithms.als [ 191ms] finished epoch 6 (|ΔP|=2.872, |ΔQ|=8.249)\n",
      "[   INFO] lenskit.algorithms.als [ 205ms] finished epoch 7 (|ΔP|=2.398, |ΔQ|=6.909)\n",
      "[   INFO] lenskit.algorithms.als [ 220ms] finished epoch 8 (|ΔP|=2.057, |ΔQ|=5.918)\n",
      "[   INFO] lenskit.algorithms.als [ 234ms] finished epoch 9 (|ΔP|=1.798, |ΔQ|=5.144)\n",
      "[   INFO] lenskit.algorithms.als [ 249ms] finished epoch 10 (|ΔP|=1.592, |ΔQ|=4.519)\n",
      "[   INFO] lenskit.algorithms.als [ 263ms] finished epoch 11 (|ΔP|=1.424, |ΔQ|=4.011)\n",
      "[   INFO] lenskit.algorithms.als [ 278ms] finished epoch 12 (|ΔP|=1.287, |ΔQ|=3.600)\n",
      "[   INFO] lenskit.algorithms.als [ 292ms] finished epoch 13 (|ΔP|=1.174, |ΔQ|=3.268)\n",
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 14 (|ΔP|=1.080, |ΔQ|=2.995)\n",
      "[   INFO] lenskit.algorithms.als [ 318ms] finished epoch 15 (|ΔP|=1.001, |ΔQ|=2.768)\n",
      "[   INFO] lenskit.algorithms.als [ 332ms] finished epoch 16 (|ΔP|=0.933, |ΔQ|=2.574)\n",
      "[   INFO] lenskit.algorithms.als [ 347ms] finished epoch 17 (|ΔP|=0.874, |ΔQ|=2.407)\n",
      "[   INFO] lenskit.algorithms.als [ 362ms] finished epoch 18 (|ΔP|=0.821, |ΔQ|=2.260)\n",
      "[   INFO] lenskit.algorithms.als [ 376ms] finished epoch 19 (|ΔP|=0.773, |ΔQ|=2.130)\n",
      "[   INFO] lenskit.algorithms.als trained model in  377ms (|P|=29.763116, |Q|=94.459451)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=20, regularization=0.1) in  392ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-u92vpizr.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.97s\n",
      "[   INFO] lenskit.batch._multi run 19: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-cj21w497.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=20, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.91s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.12s\n",
      "[   INFO] lenskit.batch._multi run 19: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 19: als.BiasedMF(features=20, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 20: als.BiasedMF(features=30, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=30, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=30, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9056 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 38ms] normalizing 671x9056 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 47ms] training biased MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als [ 63ms] finished epoch 0 (|ΔP|=33.405, |ΔQ|=127.177)\n",
      "[   INFO] lenskit.algorithms.als [ 80ms] finished epoch 1 (|ΔP|=12.561, |ΔQ|=53.704)\n",
      "[   INFO] lenskit.algorithms.als [ 98ms] finished epoch 2 (|ΔP|=9.186, |ΔQ|=27.274)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 3 (|ΔP|=6.296, |ΔQ|=17.418)\n",
      "[   INFO] lenskit.algorithms.als [ 136ms] finished epoch 4 (|ΔP|=4.582, |ΔQ|=12.845)\n",
      "[   INFO] lenskit.algorithms.als [ 154ms] finished epoch 5 (|ΔP|=3.511, |ΔQ|=9.957)\n",
      "[   INFO] lenskit.algorithms.als [ 173ms] finished epoch 6 (|ΔP|=2.806, |ΔQ|=7.989)\n",
      "[   INFO] lenskit.algorithms.als [ 192ms] finished epoch 7 (|ΔP|=2.315, |ΔQ|=6.584)\n",
      "[   INFO] lenskit.algorithms.als [ 211ms] finished epoch 8 (|ΔP|=1.957, |ΔQ|=5.542)\n",
      "[   INFO] lenskit.algorithms.als [ 230ms] finished epoch 9 (|ΔP|=1.687, |ΔQ|=4.744)\n",
      "[   INFO] lenskit.algorithms.als [ 249ms] finished epoch 10 (|ΔP|=1.478, |ΔQ|=4.124)\n",
      "[   INFO] lenskit.algorithms.als [ 267ms] finished epoch 11 (|ΔP|=1.312, |ΔQ|=3.634)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 287ms] finished epoch 12 (|ΔP|=1.176, |ΔQ|=3.241)\n",
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 13 (|ΔP|=1.063, |ΔQ|=2.917)\n",
      "[   INFO] lenskit.algorithms.als [ 323ms] finished epoch 14 (|ΔP|=0.966, |ΔQ|=2.645)\n",
      "[   INFO] lenskit.algorithms.als [ 342ms] finished epoch 15 (|ΔP|=0.883, |ΔQ|=2.411)\n",
      "[   INFO] lenskit.algorithms.als [ 360ms] finished epoch 16 (|ΔP|=0.811, |ΔQ|=2.208)\n",
      "[   INFO] lenskit.algorithms.als [ 379ms] finished epoch 17 (|ΔP|=0.748, |ΔQ|=2.031)\n",
      "[   INFO] lenskit.algorithms.als [ 395ms] finished epoch 18 (|ΔP|=0.693, |ΔQ|=1.876)\n",
      "[   INFO] lenskit.algorithms.als [ 413ms] finished epoch 19 (|ΔP|=0.644, |ΔQ|=1.739)\n",
      "[   INFO] lenskit.algorithms.als trained model in  414ms (|P|=30.828555, |Q|=94.594339)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=30, regularization=0.1) in  429ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-l7511vib.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 20: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-1ecmmz__.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=30, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.09s\n",
      "[   INFO] lenskit.batch._multi run 20: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 20: als.BiasedMF(features=30, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 21: als.BiasedMF(features=40, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=40, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=40, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9056 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 37ms] normalizing 671x9056 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 47ms] training biased MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als [ 69ms] finished epoch 0 (|ΔP|=33.989, |ΔQ|=128.492)\n",
      "[   INFO] lenskit.algorithms.als [ 90ms] finished epoch 1 (|ΔP|=12.852, |ΔQ|=54.033)\n",
      "[   INFO] lenskit.algorithms.als [ 113ms] finished epoch 2 (|ΔP|=9.362, |ΔQ|=26.681)\n",
      "[   INFO] lenskit.algorithms.als [ 138ms] finished epoch 3 (|ΔP|=6.198, |ΔQ|=17.205)\n",
      "[   INFO] lenskit.algorithms.als [ 162ms] finished epoch 4 (|ΔP|=4.391, |ΔQ|=12.593)\n",
      "[   INFO] lenskit.algorithms.als [ 187ms] finished epoch 5 (|ΔP|=3.300, |ΔQ|=9.644)\n",
      "[   INFO] lenskit.algorithms.als [ 211ms] finished epoch 6 (|ΔP|=2.605, |ΔQ|=7.686)\n",
      "[   INFO] lenskit.algorithms.als [ 237ms] finished epoch 7 (|ΔP|=2.133, |ΔQ|=6.310)\n",
      "[   INFO] lenskit.algorithms.als [ 262ms] finished epoch 8 (|ΔP|=1.791, |ΔQ|=5.293)\n",
      "[   INFO] lenskit.algorithms.als [ 285ms] finished epoch 9 (|ΔP|=1.532, |ΔQ|=4.517)\n",
      "[   INFO] lenskit.algorithms.als [ 309ms] finished epoch 10 (|ΔP|=1.329, |ΔQ|=3.911)\n",
      "[   INFO] lenskit.algorithms.als [ 332ms] finished epoch 11 (|ΔP|=1.168, |ΔQ|=3.426)\n",
      "[   INFO] lenskit.algorithms.als [ 356ms] finished epoch 12 (|ΔP|=1.037, |ΔQ|=3.032)\n",
      "[   INFO] lenskit.algorithms.als [ 377ms] finished epoch 13 (|ΔP|=0.930, |ΔQ|=2.710)\n",
      "[   INFO] lenskit.algorithms.als [ 398ms] finished epoch 14 (|ΔP|=0.842, |ΔQ|=2.444)\n",
      "[   INFO] lenskit.algorithms.als [ 422ms] finished epoch 15 (|ΔP|=0.767, |ΔQ|=2.216)\n",
      "[   INFO] lenskit.algorithms.als [ 446ms] finished epoch 16 (|ΔP|=0.702, |ΔQ|=2.017)\n",
      "[   INFO] lenskit.algorithms.als [ 467ms] finished epoch 17 (|ΔP|=0.645, |ΔQ|=1.841)\n",
      "[   INFO] lenskit.algorithms.als [ 489ms] finished epoch 18 (|ΔP|=0.594, |ΔQ|=1.684)\n",
      "[   INFO] lenskit.algorithms.als [ 513ms] finished epoch 19 (|ΔP|=0.549, |ΔQ|=1.545)\n",
      "[   INFO] lenskit.algorithms.als trained model in  516ms (|P|=31.327223, |Q|=94.660490)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=40, regularization=0.1) in  532ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-q1n41ihm.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.86s\n",
      "[   INFO] lenskit.batch._multi run 21: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-13pzc4w4.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=40, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.83s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.03s\n",
      "[   INFO] lenskit.batch._multi run 21: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 21: als.BiasedMF(features=40, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 22: als.BiasedMF(features=50, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=50, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=50, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9056 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 40ms] normalizing 671x9056 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 52ms] training biased MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als [ 79ms] finished epoch 0 (|ΔP|=34.317, |ΔQ|=129.361)\n",
      "[   INFO] lenskit.algorithms.als [ 107ms] finished epoch 1 (|ΔP|=13.053, |ΔQ|=54.269)\n",
      "[   INFO] lenskit.algorithms.als [ 132ms] finished epoch 2 (|ΔP|=9.375, |ΔQ|=26.523)\n",
      "[   INFO] lenskit.algorithms.als [ 161ms] finished epoch 3 (|ΔP|=6.160, |ΔQ|=16.806)\n",
      "[   INFO] lenskit.algorithms.als [ 191ms] finished epoch 4 (|ΔP|=4.415, |ΔQ|=12.262)\n",
      "[   INFO] lenskit.algorithms.als [ 217ms] finished epoch 5 (|ΔP|=3.351, |ΔQ|=9.409)\n",
      "[   INFO] lenskit.algorithms.als [ 245ms] finished epoch 6 (|ΔP|=2.635, |ΔQ|=7.482)\n",
      "[   INFO] lenskit.algorithms.als [ 276ms] finished epoch 7 (|ΔP|=2.127, |ΔQ|=6.115)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 8 (|ΔP|=1.757, |ΔQ|=5.107)\n",
      "[   INFO] lenskit.algorithms.als [ 332ms] finished epoch 9 (|ΔP|=1.480, |ΔQ|=4.337)\n",
      "[   INFO] lenskit.algorithms.als [ 356ms] finished epoch 10 (|ΔP|=1.268, |ΔQ|=3.731)\n",
      "[   INFO] lenskit.algorithms.als [ 385ms] finished epoch 11 (|ΔP|=1.103, |ΔQ|=3.246)\n",
      "[   INFO] lenskit.algorithms.als [ 413ms] finished epoch 12 (|ΔP|=0.972, |ΔQ|=2.851)\n",
      "[   INFO] lenskit.algorithms.als [ 442ms] finished epoch 13 (|ΔP|=0.865, |ΔQ|=2.524)\n",
      "[   INFO] lenskit.algorithms.als [ 470ms] finished epoch 14 (|ΔP|=0.777, |ΔQ|=2.251)\n",
      "[   INFO] lenskit.algorithms.als [ 494ms] finished epoch 15 (|ΔP|=0.703, |ΔQ|=2.022)\n",
      "[   INFO] lenskit.algorithms.als [ 523ms] finished epoch 16 (|ΔP|=0.641, |ΔQ|=1.826)\n",
      "[   INFO] lenskit.algorithms.als [ 552ms] finished epoch 17 (|ΔP|=0.588, |ΔQ|=1.656)\n",
      "[   INFO] lenskit.algorithms.als [ 580ms] finished epoch 18 (|ΔP|=0.541, |ΔQ|=1.508)\n",
      "[   INFO] lenskit.algorithms.als [ 608ms] finished epoch 19 (|ΔP|=0.501, |ΔQ|=1.379)\n",
      "[   INFO] lenskit.algorithms.als trained model in  611ms (|P|=31.567742, |Q|=94.697990)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=50, regularization=0.1) in  627ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-svx709_b.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.97s\n",
      "[   INFO] lenskit.batch._multi run 22: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-e06n3ya6.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=50, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.91s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.11s\n",
      "[   INFO] lenskit.batch._multi run 22: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 22: als.BiasedMF(features=50, regularization=0.1) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 23: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=20, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=20, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 12ms] training implicit MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9056 items\n",
      "[   INFO] lenskit.algorithms.als [ 62ms] finished epoch 0 (|ΔP|=3357.915, |ΔQ|=8.710)\n",
      "[   INFO] lenskit.algorithms.als [ 111ms] finished epoch 1 (|ΔP|=181.985, |ΔQ|=3.579)\n",
      "[   INFO] lenskit.algorithms.als [ 159ms] finished epoch 2 (|ΔP|=73.876, |ΔQ|=2.344)\n",
      "[   INFO] lenskit.algorithms.als [ 208ms] finished epoch 3 (|ΔP|=68.983, |ΔQ|=1.764)\n",
      "[   INFO] lenskit.algorithms.als [ 256ms] finished epoch 4 (|ΔP|=84.321, |ΔQ|=1.509)\n",
      "[   INFO] lenskit.algorithms.als [ 304ms] finished epoch 5 (|ΔP|=81.711, |ΔQ|=1.296)\n",
      "[   INFO] lenskit.algorithms.als [ 352ms] finished epoch 6 (|ΔP|=81.461, |ΔQ|=1.112)\n",
      "[   INFO] lenskit.algorithms.als [ 399ms] finished epoch 7 (|ΔP|=71.520, |ΔQ|=0.957)\n",
      "[   INFO] lenskit.algorithms.als [ 448ms] finished epoch 8 (|ΔP|=75.259, |ΔQ|=0.858)\n",
      "[   INFO] lenskit.algorithms.als [ 496ms] finished epoch 9 (|ΔP|=65.188, |ΔQ|=0.749)\n",
      "[   INFO] lenskit.algorithms.als [ 543ms] finished epoch 10 (|ΔP|=61.624, |ΔQ|=0.689)\n",
      "[   INFO] lenskit.algorithms.als [ 591ms] finished epoch 11 (|ΔP|=65.319, |ΔQ|=0.644)\n",
      "[   INFO] lenskit.algorithms.als [ 640ms] finished epoch 12 (|ΔP|=56.871, |ΔQ|=0.582)\n",
      "[   INFO] lenskit.algorithms.als [ 688ms] finished epoch 13 (|ΔP|=56.329, |ΔQ|=0.559)\n",
      "[   INFO] lenskit.algorithms.als [ 735ms] finished epoch 14 (|ΔP|=65.649, |ΔQ|=0.535)\n",
      "[   INFO] lenskit.algorithms.als [ 782ms] finished epoch 15 (|ΔP|=52.884, |ΔQ|=0.493)\n",
      "[   INFO] lenskit.algorithms.als [ 830ms] finished epoch 16 (|ΔP|=50.796, |ΔQ|=0.464)\n",
      "[   INFO] lenskit.algorithms.als [ 877ms] finished epoch 17 (|ΔP|=49.582, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [ 925ms] finished epoch 18 (|ΔP|=52.831, |ΔQ|=0.425)\n",
      "[   INFO] lenskit.algorithms.als [ 972ms] finished epoch 19 (|ΔP|=51.524, |ΔQ|=0.413)\n",
      "[   INFO] lenskit.algorithms.als [ 973ms] finished training model with 20 features (|P|=2594.843938, |Q|=9.685295)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=20, reg=0.1, w=40) in  986ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-77147wlo.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 23: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-hhgfvnia.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=20, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.10s\n",
      "[   INFO] lenskit.batch._multi run 23: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 23: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 24: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=30, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=30, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 14ms] training implicit MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9056 items\n",
      "[   INFO] lenskit.algorithms.als [ 67ms] finished epoch 0 (|ΔP|=3788.065, |ΔQ|=8.599)\n",
      "[   INFO] lenskit.algorithms.als [ 119ms] finished epoch 1 (|ΔP|=184.942, |ΔQ|=3.474)\n",
      "[   INFO] lenskit.algorithms.als [ 170ms] finished epoch 2 (|ΔP|=99.117, |ΔQ|=2.289)\n",
      "[   INFO] lenskit.algorithms.als [ 221ms] finished epoch 3 (|ΔP|=100.120, |ΔQ|=1.767)\n",
      "[   INFO] lenskit.algorithms.als [ 272ms] finished epoch 4 (|ΔP|=107.634, |ΔQ|=1.427)\n",
      "[   INFO] lenskit.algorithms.als [ 322ms] finished epoch 5 (|ΔP|=102.516, |ΔQ|=1.165)\n",
      "[   INFO] lenskit.algorithms.als [ 373ms] finished epoch 6 (|ΔP|=100.892, |ΔQ|=0.982)\n",
      "[   INFO] lenskit.algorithms.als [ 424ms] finished epoch 7 (|ΔP|=83.033, |ΔQ|=0.834)\n",
      "[   INFO] lenskit.algorithms.als [ 476ms] finished epoch 8 (|ΔP|=83.866, |ΔQ|=0.759)\n",
      "[   INFO] lenskit.algorithms.als [ 528ms] finished epoch 9 (|ΔP|=68.626, |ΔQ|=0.655)\n",
      "[   INFO] lenskit.algorithms.als [ 580ms] finished epoch 10 (|ΔP|=82.353, |ΔQ|=0.644)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 632ms] finished epoch 11 (|ΔP|=69.467, |ΔQ|=0.568)\n",
      "[   INFO] lenskit.algorithms.als [ 683ms] finished epoch 12 (|ΔP|=79.841, |ΔQ|=0.570)\n",
      "[   INFO] lenskit.algorithms.als [ 735ms] finished epoch 13 (|ΔP|=64.893, |ΔQ|=0.504)\n",
      "[   INFO] lenskit.algorithms.als [ 786ms] finished epoch 14 (|ΔP|=105.060, |ΔQ|=0.550)\n",
      "[   INFO] lenskit.algorithms.als [ 837ms] finished epoch 15 (|ΔP|=57.707, |ΔQ|=0.453)\n",
      "[   INFO] lenskit.algorithms.als [ 888ms] finished epoch 16 (|ΔP|=78.045, |ΔQ|=0.488)\n",
      "[   INFO] lenskit.algorithms.als [ 940ms] finished epoch 17 (|ΔP|=50.216, |ΔQ|=0.406)\n",
      "[   INFO] lenskit.algorithms.als [ 992ms] finished epoch 18 (|ΔP|=82.333, |ΔQ|=0.452)\n",
      "[   INFO] lenskit.algorithms.als [1.05s] finished epoch 19 (|ΔP|=48.702, |ΔQ|=0.375)\n",
      "[   INFO] lenskit.algorithms.als [1.05s] finished training model with 30 features (|P|=2806.043472, |Q|=9.975942)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=30, reg=0.1, w=40) in 1.07s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-8mv4_f9p.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 24: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-cghi931b.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=30, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.85s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.05s\n",
      "[   INFO] lenskit.batch._multi run 24: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 24: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 25: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=40, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=40, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 15ms] training implicit MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9056 items\n",
      "[   INFO] lenskit.algorithms.als [ 66ms] finished epoch 0 (|ΔP|=4092.606, |ΔQ|=7.808)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 1 (|ΔP|=223.575, |ΔQ|=3.012)\n",
      "[   INFO] lenskit.algorithms.als [ 170ms] finished epoch 2 (|ΔP|=148.022, |ΔQ|=2.026)\n",
      "[   INFO] lenskit.algorithms.als [ 223ms] finished epoch 3 (|ΔP|=178.204, |ΔQ|=1.554)\n",
      "[   INFO] lenskit.algorithms.als [ 276ms] finished epoch 4 (|ΔP|=169.362, |ΔQ|=1.191)\n",
      "[   INFO] lenskit.algorithms.als [ 329ms] finished epoch 5 (|ΔP|=146.490, |ΔQ|=0.962)\n",
      "[   INFO] lenskit.algorithms.als [ 382ms] finished epoch 6 (|ΔP|=140.628, |ΔQ|=0.823)\n",
      "[   INFO] lenskit.algorithms.als [ 435ms] finished epoch 7 (|ΔP|=121.566, |ΔQ|=0.711)\n",
      "[   INFO] lenskit.algorithms.als [ 487ms] finished epoch 8 (|ΔP|=105.469, |ΔQ|=0.625)\n",
      "[   INFO] lenskit.algorithms.als [ 541ms] finished epoch 9 (|ΔP|=106.792, |ΔQ|=0.587)\n",
      "[   INFO] lenskit.algorithms.als [ 595ms] finished epoch 10 (|ΔP|=105.799, |ΔQ|=0.540)\n",
      "[   INFO] lenskit.algorithms.als [ 648ms] finished epoch 11 (|ΔP|=109.223, |ΔQ|=0.522)\n",
      "[   INFO] lenskit.algorithms.als [ 701ms] finished epoch 12 (|ΔP|=97.034, |ΔQ|=0.469)\n",
      "[   INFO] lenskit.algorithms.als [ 754ms] finished epoch 13 (|ΔP|=99.933, |ΔQ|=0.467)\n",
      "[   INFO] lenskit.algorithms.als [ 806ms] finished epoch 14 (|ΔP|=86.103, |ΔQ|=0.418)\n",
      "[   INFO] lenskit.algorithms.als [ 860ms] finished epoch 15 (|ΔP|=108.292, |ΔQ|=0.436)\n",
      "[   INFO] lenskit.algorithms.als [ 913ms] finished epoch 16 (|ΔP|=72.599, |ΔQ|=0.370)\n",
      "[   INFO] lenskit.algorithms.als [ 970ms] finished epoch 17 (|ΔP|=98.976, |ΔQ|=0.396)\n",
      "[   INFO] lenskit.algorithms.als [1.02s] finished epoch 18 (|ΔP|=73.355, |ΔQ|=0.346)\n",
      "[   INFO] lenskit.algorithms.als [1.08s] finished epoch 19 (|ΔP|=98.873, |ΔQ|=0.390)\n",
      "[   INFO] lenskit.algorithms.als [1.08s] finished training model with 40 features (|P|=2658.712288, |Q|=8.539329)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=40, reg=0.1, w=40) in 1.09s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-nyem282k.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.91s\n",
      "[   INFO] lenskit.batch._multi run 25: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-nxcu4s0t.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=40, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.84s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.05s\n",
      "[   INFO] lenskit.batch._multi run 25: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 25: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 26: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=50, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=50, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 18ms] training implicit MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9056 items\n",
      "[   INFO] lenskit.algorithms.als [ 74ms] finished epoch 0 (|ΔP|=4311.068, |ΔQ|=7.565)\n",
      "[   INFO] lenskit.algorithms.als [ 128ms] finished epoch 1 (|ΔP|=231.260, |ΔQ|=2.910)\n",
      "[   INFO] lenskit.algorithms.als [ 183ms] finished epoch 2 (|ΔP|=200.886, |ΔQ|=1.865)\n",
      "[   INFO] lenskit.algorithms.als [ 238ms] finished epoch 3 (|ΔP|=232.445, |ΔQ|=1.397)\n",
      "[   INFO] lenskit.algorithms.als [ 292ms] finished epoch 4 (|ΔP|=198.132, |ΔQ|=1.074)\n",
      "[   INFO] lenskit.algorithms.als [ 347ms] finished epoch 5 (|ΔP|=169.233, |ΔQ|=0.858)\n",
      "[   INFO] lenskit.algorithms.als [ 404ms] finished epoch 6 (|ΔP|=150.187, |ΔQ|=0.742)\n",
      "[   INFO] lenskit.algorithms.als [ 459ms] finished epoch 7 (|ΔP|=139.323, |ΔQ|=0.651)\n",
      "[   INFO] lenskit.algorithms.als [ 514ms] finished epoch 8 (|ΔP|=140.788, |ΔQ|=0.593)\n",
      "[   INFO] lenskit.algorithms.als [ 569ms] finished epoch 9 (|ΔP|=120.490, |ΔQ|=0.533)\n",
      "[   INFO] lenskit.algorithms.als [ 624ms] finished epoch 10 (|ΔP|=115.518, |ΔQ|=0.491)\n",
      "[   INFO] lenskit.algorithms.als [ 679ms] finished epoch 11 (|ΔP|=112.176, |ΔQ|=0.460)\n",
      "[   INFO] lenskit.algorithms.als [ 734ms] finished epoch 12 (|ΔP|=107.829, |ΔQ|=0.432)\n",
      "[   INFO] lenskit.algorithms.als [ 789ms] finished epoch 13 (|ΔP|=107.469, |ΔQ|=0.410)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 867ms] finished epoch 14 (|ΔP|=97.138, |ΔQ|=0.395)\n",
      "[   INFO] lenskit.algorithms.als [ 939ms] finished epoch 15 (|ΔP|=104.980, |ΔQ|=0.384)\n",
      "[   INFO] lenskit.algorithms.als [ 999ms] finished epoch 16 (|ΔP|=87.470, |ΔQ|=0.352)\n",
      "[   INFO] lenskit.algorithms.als [1.05s] finished epoch 17 (|ΔP|=102.041, |ΔQ|=0.357)\n",
      "[   INFO] lenskit.algorithms.als [1.11s] finished epoch 18 (|ΔP|=88.463, |ΔQ|=0.337)\n",
      "[   INFO] lenskit.algorithms.als [1.16s] finished epoch 19 (|ΔP|=89.352, |ΔQ|=0.331)\n",
      "[   INFO] lenskit.algorithms.als [1.17s] finished training model with 50 features (|P|=2621.603224, |Q|=8.521143)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=50, reg=0.1, w=40) in 1.18s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-b30jtcwd.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.99s\n",
      "[   INFO] lenskit.batch._multi run 26: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-0d04mu3w.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=50, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 2.05s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.28s\n",
      "[   INFO] lenskit.batch._multi run 26: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 26: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 27: Popular on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi training algorithm Popular on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm Popular in  14ms\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for Popular\n",
      "[   INFO] lenskit.sharing persisting Popular to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-642blguk.bpk\n",
      "[   INFO] binpickle.write pickled 967 bytes with 7 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with Popular for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in  772ms\n",
      "[   INFO] lenskit.batch._multi generated recommendations in  846ms\n",
      "[   INFO] lenskit.batch._multi run 27: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 27: Popular on ML-Small:3\n",
      "[   INFO] lenskit.batch._multi starting run 28: als.BiasedMF(features=20, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=20, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=20, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 37ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 48ms] training biased MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als [ 63ms] finished epoch 0 (|ΔP|=32.464, |ΔQ|=124.546)\n",
      "[   INFO] lenskit.algorithms.als [ 75ms] finished epoch 1 (|ΔP|=12.140, |ΔQ|=52.024)\n",
      "[   INFO] lenskit.algorithms.als [ 87ms] finished epoch 2 (|ΔP|=8.842, |ΔQ|=26.757)\n",
      "[   INFO] lenskit.algorithms.als [ 100ms] finished epoch 3 (|ΔP|=6.100, |ΔQ|=17.223)\n",
      "[   INFO] lenskit.algorithms.als [ 113ms] finished epoch 4 (|ΔP|=4.500, |ΔQ|=12.806)\n",
      "[   INFO] lenskit.algorithms.als [ 126ms] finished epoch 5 (|ΔP|=3.538, |ΔQ|=10.055)\n",
      "[   INFO] lenskit.algorithms.als [ 140ms] finished epoch 6 (|ΔP|=2.924, |ΔQ|=8.199)\n",
      "[   INFO] lenskit.algorithms.als [ 155ms] finished epoch 7 (|ΔP|=2.485, |ΔQ|=6.874)\n",
      "[   INFO] lenskit.algorithms.als [ 167ms] finished epoch 8 (|ΔP|=2.141, |ΔQ|=5.877)\n",
      "[   INFO] lenskit.algorithms.als [ 181ms] finished epoch 9 (|ΔP|=1.858, |ΔQ|=5.099)\n",
      "[   INFO] lenskit.algorithms.als [ 194ms] finished epoch 10 (|ΔP|=1.623, |ΔQ|=4.479)\n",
      "[   INFO] lenskit.algorithms.als [ 207ms] finished epoch 11 (|ΔP|=1.429, |ΔQ|=3.978)\n",
      "[   INFO] lenskit.algorithms.als [ 222ms] finished epoch 12 (|ΔP|=1.270, |ΔQ|=3.568)\n",
      "[   INFO] lenskit.algorithms.als [ 236ms] finished epoch 13 (|ΔP|=1.140, |ΔQ|=3.229)\n",
      "[   INFO] lenskit.algorithms.als [ 250ms] finished epoch 14 (|ΔP|=1.034, |ΔQ|=2.948)\n",
      "[   INFO] lenskit.algorithms.als [ 265ms] finished epoch 15 (|ΔP|=0.946, |ΔQ|=2.712)\n",
      "[   INFO] lenskit.algorithms.als [ 279ms] finished epoch 16 (|ΔP|=0.873, |ΔQ|=2.514)\n",
      "[   INFO] lenskit.algorithms.als [ 292ms] finished epoch 17 (|ΔP|=0.811, |ΔQ|=2.344)\n",
      "[   INFO] lenskit.algorithms.als [ 306ms] finished epoch 18 (|ΔP|=0.758, |ΔQ|=2.198)\n",
      "[   INFO] lenskit.algorithms.als [ 320ms] finished epoch 19 (|ΔP|=0.712, |ΔQ|=2.068)\n",
      "[   INFO] lenskit.algorithms.als trained model in  322ms (|P|=29.735833, |Q|=94.434018)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=20, regularization=0.1) in  337ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-0pfw3yto.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 2.04s\n",
      "[   INFO] lenskit.batch._multi run 28: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-hvfd9b1a.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=20, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.10s\n",
      "[   INFO] lenskit.batch._multi run 28: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 28: als.BiasedMF(features=20, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 29: als.BiasedMF(features=30, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=30, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=30, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 36ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 45ms] training biased MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als [ 62ms] finished epoch 0 (|ΔP|=33.149, |ΔQ|=126.683)\n",
      "[   INFO] lenskit.algorithms.als [ 80ms] finished epoch 1 (|ΔP|=12.531, |ΔQ|=54.027)\n",
      "[   INFO] lenskit.algorithms.als [ 98ms] finished epoch 2 (|ΔP|=9.214, |ΔQ|=27.636)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 3 (|ΔP|=6.197, |ΔQ|=17.192)\n",
      "[   INFO] lenskit.algorithms.als [ 135ms] finished epoch 4 (|ΔP|=4.445, |ΔQ|=12.560)\n",
      "[   INFO] lenskit.algorithms.als [ 153ms] finished epoch 5 (|ΔP|=3.410, |ΔQ|=9.730)\n",
      "[   INFO] lenskit.algorithms.als [ 172ms] finished epoch 6 (|ΔP|=2.751, |ΔQ|=7.807)\n",
      "[   INFO] lenskit.algorithms.als [ 191ms] finished epoch 7 (|ΔP|=2.293, |ΔQ|=6.434)\n",
      "[   INFO] lenskit.algorithms.als [ 208ms] finished epoch 8 (|ΔP|=1.952, |ΔQ|=5.428)\n",
      "[   INFO] lenskit.algorithms.als [ 227ms] finished epoch 9 (|ΔP|=1.689, |ΔQ|=4.677)\n",
      "[   INFO] lenskit.algorithms.als [ 246ms] finished epoch 10 (|ΔP|=1.482, |ΔQ|=4.101)\n",
      "[   INFO] lenskit.algorithms.als [ 264ms] finished epoch 11 (|ΔP|=1.317, |ΔQ|=3.647)\n",
      "[   INFO] lenskit.algorithms.als [ 283ms] finished epoch 12 (|ΔP|=1.184, |ΔQ|=3.277)\n",
      "[   INFO] lenskit.algorithms.als [ 302ms] finished epoch 13 (|ΔP|=1.074, |ΔQ|=2.966)\n",
      "[   INFO] lenskit.algorithms.als [ 320ms] finished epoch 14 (|ΔP|=0.981, |ΔQ|=2.696)\n",
      "[   INFO] lenskit.algorithms.als [ 339ms] finished epoch 15 (|ΔP|=0.900, |ΔQ|=2.459)\n",
      "[   INFO] lenskit.algorithms.als [ 357ms] finished epoch 16 (|ΔP|=0.829, |ΔQ|=2.249)\n",
      "[   INFO] lenskit.algorithms.als [ 375ms] finished epoch 17 (|ΔP|=0.765, |ΔQ|=2.065)\n",
      "[   INFO] lenskit.algorithms.als [ 394ms] finished epoch 18 (|ΔP|=0.709, |ΔQ|=1.903)\n",
      "[   INFO] lenskit.algorithms.als [ 411ms] finished epoch 19 (|ΔP|=0.659, |ΔQ|=1.762)\n",
      "[   INFO] lenskit.algorithms.als trained model in  412ms (|P|=30.778758, |Q|=94.562917)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=30, regularization=0.1) in  427ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-dtd2xilv.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.93s\n",
      "[   INFO] lenskit.batch._multi run 29: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-6kb_y4oo.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=30, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.84s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.07s\n",
      "[   INFO] lenskit.batch._multi run 29: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 29: als.BiasedMF(features=30, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 30: als.BiasedMF(features=40, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=40, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=40, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 41ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 52ms] training biased MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als [ 73ms] finished epoch 0 (|ΔP|=33.861, |ΔQ|=128.130)\n",
      "[   INFO] lenskit.algorithms.als [ 96ms] finished epoch 1 (|ΔP|=12.860, |ΔQ|=54.030)\n",
      "[   INFO] lenskit.algorithms.als [ 120ms] finished epoch 2 (|ΔP|=9.463, |ΔQ|=27.509)\n",
      "[   INFO] lenskit.algorithms.als [ 142ms] finished epoch 3 (|ΔP|=6.230, |ΔQ|=17.281)\n",
      "[   INFO] lenskit.algorithms.als [ 166ms] finished epoch 4 (|ΔP|=4.419, |ΔQ|=12.543)\n",
      "[   INFO] lenskit.algorithms.als [ 190ms] finished epoch 5 (|ΔP|=3.367, |ΔQ|=9.627)\n",
      "[   INFO] lenskit.algorithms.als [ 215ms] finished epoch 6 (|ΔP|=2.688, |ΔQ|=7.703)\n",
      "[   INFO] lenskit.algorithms.als [ 236ms] finished epoch 7 (|ΔP|=2.212, |ΔQ|=6.337)\n",
      "[   INFO] lenskit.algorithms.als [ 259ms] finished epoch 8 (|ΔP|=1.857, |ΔQ|=5.311)\n",
      "[   INFO] lenskit.algorithms.als [ 283ms] finished epoch 9 (|ΔP|=1.583, |ΔQ|=4.515)\n",
      "[   INFO] lenskit.algorithms.als [ 307ms] finished epoch 10 (|ΔP|=1.367, |ΔQ|=3.886)\n",
      "[   INFO] lenskit.algorithms.als [ 332ms] finished epoch 11 (|ΔP|=1.193, |ΔQ|=3.385)\n",
      "[   INFO] lenskit.algorithms.als [ 359ms] finished epoch 12 (|ΔP|=1.052, |ΔQ|=2.984)\n",
      "[   INFO] lenskit.algorithms.als [ 384ms] finished epoch 13 (|ΔP|=0.936, |ΔQ|=2.659)\n",
      "[   INFO] lenskit.algorithms.als [ 408ms] finished epoch 14 (|ΔP|=0.840, |ΔQ|=2.392)\n",
      "[   INFO] lenskit.algorithms.als [ 431ms] finished epoch 15 (|ΔP|=0.760, |ΔQ|=2.168)\n",
      "[   INFO] lenskit.algorithms.als [ 455ms] finished epoch 16 (|ΔP|=0.693, |ΔQ|=1.977)\n",
      "[   INFO] lenskit.algorithms.als [ 478ms] finished epoch 17 (|ΔP|=0.635, |ΔQ|=1.811)\n",
      "[   INFO] lenskit.algorithms.als [ 501ms] finished epoch 18 (|ΔP|=0.586, |ΔQ|=1.666)\n",
      "[   INFO] lenskit.algorithms.als [ 525ms] finished epoch 19 (|ΔP|=0.542, |ΔQ|=1.538)\n",
      "[   INFO] lenskit.algorithms.als trained model in  528ms (|P|=31.278303, |Q|=94.632456)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=40, regularization=0.1) in  544ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-004ltxjh.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.96s\n",
      "[   INFO] lenskit.batch._multi run 30: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-i1xhpqu0.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=40, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.97s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.19s\n",
      "[   INFO] lenskit.batch._multi run 30: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 30: als.BiasedMF(features=40, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 31: als.BiasedMF(features=50, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=50, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=50, regularization=0.1) on 99334 ratings\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9059 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 40ms] normalizing 671x9059 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 54ms] training biased MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als [ 81ms] finished epoch 0 (|ΔP|=34.181, |ΔQ|=128.832)\n",
      "[   INFO] lenskit.algorithms.als [ 111ms] finished epoch 1 (|ΔP|=13.037, |ΔQ|=54.526)\n",
      "[   INFO] lenskit.algorithms.als [ 135ms] finished epoch 2 (|ΔP|=9.395, |ΔQ|=27.100)\n",
      "[   INFO] lenskit.algorithms.als [ 165ms] finished epoch 3 (|ΔP|=6.127, |ΔQ|=17.173)\n",
      "[   INFO] lenskit.algorithms.als [ 194ms] finished epoch 4 (|ΔP|=4.313, |ΔQ|=12.515)\n",
      "[   INFO] lenskit.algorithms.als [ 219ms] finished epoch 5 (|ΔP|=3.226, |ΔQ|=9.601)\n",
      "[   INFO] lenskit.algorithms.als [ 249ms] finished epoch 6 (|ΔP|=2.527, |ΔQ|=7.652)\n",
      "[   INFO] lenskit.algorithms.als [ 279ms] finished epoch 7 (|ΔP|=2.046, |ΔQ|=6.273)\n",
      "[   INFO] lenskit.algorithms.als [ 305ms] finished epoch 8 (|ΔP|=1.696, |ΔQ|=5.243)\n",
      "[   INFO] lenskit.algorithms.als [ 333ms] finished epoch 9 (|ΔP|=1.431, |ΔQ|=4.436)\n",
      "[   INFO] lenskit.algorithms.als [ 362ms] finished epoch 10 (|ΔP|=1.225, |ΔQ|=3.791)\n",
      "[   INFO] lenskit.algorithms.als [ 390ms] finished epoch 11 (|ΔP|=1.063, |ΔQ|=3.272)\n",
      "[   INFO] lenskit.algorithms.als [ 419ms] finished epoch 12 (|ΔP|=0.934, |ΔQ|=2.855)\n",
      "[   INFO] lenskit.algorithms.als [ 448ms] finished epoch 13 (|ΔP|=0.829, |ΔQ|=2.519)\n",
      "[   INFO] lenskit.algorithms.als [ 477ms] finished epoch 14 (|ΔP|=0.743, |ΔQ|=2.242)\n",
      "[   INFO] lenskit.algorithms.als [ 508ms] finished epoch 15 (|ΔP|=0.672, |ΔQ|=2.008)\n",
      "[   INFO] lenskit.algorithms.als [ 538ms] finished epoch 16 (|ΔP|=0.613, |ΔQ|=1.810)\n",
      "[   INFO] lenskit.algorithms.als [ 567ms] finished epoch 17 (|ΔP|=0.563, |ΔQ|=1.643)\n",
      "[   INFO] lenskit.algorithms.als [ 597ms] finished epoch 18 (|ΔP|=0.520, |ΔQ|=1.501)\n",
      "[   INFO] lenskit.algorithms.als [ 627ms] finished epoch 19 (|ΔP|=0.482, |ΔQ|=1.379)\n",
      "[   INFO] lenskit.algorithms.als trained model in  629ms (|P|=31.510081, |Q|=94.653882)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=50, regularization=0.1) in  646ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-2z8g23xf.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 31: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-zuoxkutj.bpk\n",
      "[   INFO] binpickle.write pickled 1340 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=50, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 2.02s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.23s\n",
      "[   INFO] lenskit.batch._multi run 31: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 31: als.BiasedMF(features=50, regularization=0.1) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 32: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=20, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=20, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 13ms] training implicit MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 61ms] finished epoch 0 (|ΔP|=3384.911, |ΔQ|=9.139)\n",
      "[   INFO] lenskit.algorithms.als [ 111ms] finished epoch 1 (|ΔP|=179.309, |ΔQ|=3.723)\n",
      "[   INFO] lenskit.algorithms.als [ 160ms] finished epoch 2 (|ΔP|=66.444, |ΔQ|=2.406)\n",
      "[   INFO] lenskit.algorithms.als [ 208ms] finished epoch 3 (|ΔP|=59.216, |ΔQ|=1.878)\n",
      "[   INFO] lenskit.algorithms.als [ 257ms] finished epoch 4 (|ΔP|=65.790, |ΔQ|=1.630)\n",
      "[   INFO] lenskit.algorithms.als [ 308ms] finished epoch 5 (|ΔP|=70.422, |ΔQ|=1.398)\n",
      "[   INFO] lenskit.algorithms.als [ 357ms] finished epoch 6 (|ΔP|=70.259, |ΔQ|=1.177)\n",
      "[   INFO] lenskit.algorithms.als [ 409ms] finished epoch 7 (|ΔP|=70.535, |ΔQ|=1.028)\n",
      "[   INFO] lenskit.algorithms.als [ 458ms] finished epoch 8 (|ΔP|=63.489, |ΔQ|=0.895)\n",
      "[   INFO] lenskit.algorithms.als [ 509ms] finished epoch 9 (|ΔP|=65.777, |ΔQ|=0.813)\n",
      "[   INFO] lenskit.algorithms.als [ 558ms] finished epoch 10 (|ΔP|=55.906, |ΔQ|=0.721)\n",
      "[   INFO] lenskit.algorithms.als [ 606ms] finished epoch 11 (|ΔP|=60.511, |ΔQ|=0.676)\n",
      "[   INFO] lenskit.algorithms.als [ 655ms] finished epoch 12 (|ΔP|=56.574, |ΔQ|=0.624)\n",
      "[   INFO] lenskit.algorithms.als [ 703ms] finished epoch 13 (|ΔP|=55.349, |ΔQ|=0.595)\n",
      "[   INFO] lenskit.algorithms.als [ 751ms] finished epoch 14 (|ΔP|=53.960, |ΔQ|=0.561)\n",
      "[   INFO] lenskit.algorithms.als [ 799ms] finished epoch 15 (|ΔP|=50.375, |ΔQ|=0.523)\n",
      "[   INFO] lenskit.algorithms.als [ 848ms] finished epoch 16 (|ΔP|=51.098, |ΔQ|=0.500)\n",
      "[   INFO] lenskit.algorithms.als [ 896ms] finished epoch 17 (|ΔP|=45.104, |ΔQ|=0.469)\n",
      "[   INFO] lenskit.algorithms.als [ 946ms] finished epoch 18 (|ΔP|=54.495, |ΔQ|=0.487)\n",
      "[   INFO] lenskit.algorithms.als [ 994ms] finished epoch 19 (|ΔP|=40.602, |ΔQ|=0.418)\n",
      "[   INFO] lenskit.algorithms.als [ 995ms] finished training model with 20 features (|P|=2711.510296, |Q|=9.958338)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=20, reg=0.1, w=40) in 1.01s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-7vhwwpbu.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.98s\n",
      "[   INFO] lenskit.batch._multi run 32: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-6l1dh9sw.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=20, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.93s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.14s\n",
      "[   INFO] lenskit.batch._multi run 32: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 32: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 33: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=30, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=30, reg=0.1, w=40) on 99334 ratings\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 14ms] training implicit MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 62ms] finished epoch 0 (|ΔP|=3805.819, |ΔQ|=8.315)\n",
      "[   INFO] lenskit.algorithms.als [ 113ms] finished epoch 1 (|ΔP|=200.755, |ΔQ|=3.272)\n",
      "[   INFO] lenskit.algorithms.als [ 164ms] finished epoch 2 (|ΔP|=107.821, |ΔQ|=2.245)\n",
      "[   INFO] lenskit.algorithms.als [ 216ms] finished epoch 3 (|ΔP|=114.832, |ΔQ|=1.754)\n",
      "[   INFO] lenskit.algorithms.als [ 269ms] finished epoch 4 (|ΔP|=128.287, |ΔQ|=1.433)\n",
      "[   INFO] lenskit.algorithms.als [ 322ms] finished epoch 5 (|ΔP|=128.031, |ΔQ|=1.156)\n",
      "[   INFO] lenskit.algorithms.als [ 377ms] finished epoch 6 (|ΔP|=106.760, |ΔQ|=0.965)\n",
      "[   INFO] lenskit.algorithms.als [ 428ms] finished epoch 7 (|ΔP|=98.827, |ΔQ|=0.829)\n",
      "[   INFO] lenskit.algorithms.als [ 482ms] finished epoch 8 (|ΔP|=98.387, |ΔQ|=0.756)\n",
      "[   INFO] lenskit.algorithms.als [ 535ms] finished epoch 9 (|ΔP|=83.666, |ΔQ|=0.657)\n",
      "[   INFO] lenskit.algorithms.als [ 587ms] finished epoch 10 (|ΔP|=94.149, |ΔQ|=0.624)\n",
      "[   INFO] lenskit.algorithms.als [ 641ms] finished epoch 11 (|ΔP|=78.380, |ΔQ|=0.558)\n",
      "[   INFO] lenskit.algorithms.als [ 693ms] finished epoch 12 (|ΔP|=88.744, |ΔQ|=0.549)\n",
      "[   INFO] lenskit.algorithms.als [ 746ms] finished epoch 13 (|ΔP|=75.823, |ΔQ|=0.490)\n",
      "[   INFO] lenskit.algorithms.als [ 799ms] finished epoch 14 (|ΔP|=89.571, |ΔQ|=0.492)\n",
      "[   INFO] lenskit.algorithms.als [ 851ms] finished epoch 15 (|ΔP|=67.314, |ΔQ|=0.442)\n",
      "[   INFO] lenskit.algorithms.als [ 905ms] finished epoch 16 (|ΔP|=82.458, |ΔQ|=0.451)\n",
      "[   INFO] lenskit.algorithms.als [ 961ms] finished epoch 17 (|ΔP|=58.195, |ΔQ|=0.392)\n",
      "[   INFO] lenskit.algorithms.als [1.01s] finished epoch 18 (|ΔP|=73.681, |ΔQ|=0.400)\n",
      "[   INFO] lenskit.algorithms.als [1.07s] finished epoch 19 (|ΔP|=59.646, |ΔQ|=0.362)\n",
      "[   INFO] lenskit.algorithms.als [1.07s] finished training model with 30 features (|P|=2689.675737, |Q|=9.299979)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=30, reg=0.1, w=40) in 1.08s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-bu3034s6.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 33: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-_48rx_al.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=30, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.80s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.00s\n",
      "[   INFO] lenskit.batch._multi run 33: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 33: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 34: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=40, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=40, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 16ms] training implicit MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 68ms] finished epoch 0 (|ΔP|=4111.694, |ΔQ|=7.796)\n",
      "[   INFO] lenskit.algorithms.als [ 118ms] finished epoch 1 (|ΔP|=212.261, |ΔQ|=3.058)\n",
      "[   INFO] lenskit.algorithms.als [ 172ms] finished epoch 2 (|ΔP|=156.573, |ΔQ|=1.997)\n",
      "[   INFO] lenskit.algorithms.als [ 225ms] finished epoch 3 (|ΔP|=174.744, |ΔQ|=1.525)\n",
      "[   INFO] lenskit.algorithms.als [ 278ms] finished epoch 4 (|ΔP|=170.051, |ΔQ|=1.188)\n",
      "[   INFO] lenskit.algorithms.als [ 330ms] finished epoch 5 (|ΔP|=136.223, |ΔQ|=0.939)\n",
      "[   INFO] lenskit.algorithms.als [ 383ms] finished epoch 6 (|ΔP|=129.221, |ΔQ|=0.807)\n",
      "[   INFO] lenskit.algorithms.als [ 438ms] finished epoch 7 (|ΔP|=123.600, |ΔQ|=0.702)\n",
      "[   INFO] lenskit.algorithms.als [ 515ms] finished epoch 8 (|ΔP|=110.418, |ΔQ|=0.627)\n",
      "[   INFO] lenskit.algorithms.als [ 577ms] finished epoch 9 (|ΔP|=111.233, |ΔQ|=0.573)\n",
      "[   INFO] lenskit.algorithms.als [ 631ms] finished epoch 10 (|ΔP|=102.674, |ΔQ|=0.536)\n",
      "[   INFO] lenskit.algorithms.als [ 685ms] finished epoch 11 (|ΔP|=98.389, |ΔQ|=0.492)\n",
      "[   INFO] lenskit.algorithms.als [ 738ms] finished epoch 12 (|ΔP|=101.211, |ΔQ|=0.467)\n",
      "[   INFO] lenskit.algorithms.als [ 791ms] finished epoch 13 (|ΔP|=91.078, |ΔQ|=0.434)\n",
      "[   INFO] lenskit.algorithms.als [ 844ms] finished epoch 14 (|ΔP|=109.555, |ΔQ|=0.439)\n",
      "[   INFO] lenskit.algorithms.als [ 904ms] finished epoch 15 (|ΔP|=79.916, |ΔQ|=0.393)\n",
      "[   INFO] lenskit.algorithms.als [ 957ms] finished epoch 16 (|ΔP|=83.805, |ΔQ|=0.386)\n",
      "[   INFO] lenskit.algorithms.als [1.01s] finished epoch 17 (|ΔP|=74.346, |ΔQ|=0.356)\n",
      "[   INFO] lenskit.algorithms.als [1.06s] finished epoch 18 (|ΔP|=79.450, |ΔQ|=0.353)\n",
      "[   INFO] lenskit.algorithms.als [1.12s] finished epoch 19 (|ΔP|=79.956, |ΔQ|=0.349)\n",
      "[   INFO] lenskit.algorithms.als [1.12s] finished training model with 40 features (|P|=2721.630024, |Q|=8.660680)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=40, reg=0.1, w=40) in 1.13s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-t8h6jxg8.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.91s\n",
      "[   INFO] lenskit.batch._multi run 34: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-wf1qx0ub.bpk\n",
      "[   INFO] binpickle.write pickled 1192 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=40, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.82s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.02s\n",
      "[   INFO] lenskit.batch._multi run 34: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 34: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 35: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=50, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=50, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 18ms] training implicit MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9059 items\n",
      "[   INFO] lenskit.algorithms.als [ 72ms] finished epoch 0 (|ΔP|=4322.730, |ΔQ|=6.940)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.algorithms.als [ 127ms] finished epoch 1 (|ΔP|=266.656, |ΔQ|=2.623)\n",
      "[   INFO] lenskit.algorithms.als [ 182ms] finished epoch 2 (|ΔP|=274.084, |ΔQ|=1.779)\n",
      "[   INFO] lenskit.algorithms.als [ 237ms] finished epoch 3 (|ΔP|=286.518, |ΔQ|=1.320)\n",
      "[   INFO] lenskit.algorithms.als [ 291ms] finished epoch 4 (|ΔP|=242.929, |ΔQ|=1.032)\n",
      "[   INFO] lenskit.algorithms.als [ 347ms] finished epoch 5 (|ΔP|=205.454, |ΔQ|=0.849)\n",
      "[   INFO] lenskit.algorithms.als [ 401ms] finished epoch 6 (|ΔP|=182.811, |ΔQ|=0.727)\n",
      "[   INFO] lenskit.algorithms.als [ 455ms] finished epoch 7 (|ΔP|=163.238, |ΔQ|=0.633)\n",
      "[   INFO] lenskit.algorithms.als [ 511ms] finished epoch 8 (|ΔP|=150.520, |ΔQ|=0.576)\n",
      "[   INFO] lenskit.algorithms.als [ 567ms] finished epoch 9 (|ΔP|=138.748, |ΔQ|=0.519)\n",
      "[   INFO] lenskit.algorithms.als [ 622ms] finished epoch 10 (|ΔP|=142.712, |ΔQ|=0.487)\n",
      "[   INFO] lenskit.algorithms.als [ 678ms] finished epoch 11 (|ΔP|=117.773, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [ 734ms] finished epoch 12 (|ΔP|=127.544, |ΔQ|=0.430)\n",
      "[   INFO] lenskit.algorithms.als [ 793ms] finished epoch 13 (|ΔP|=119.131, |ΔQ|=0.399)\n",
      "[   INFO] lenskit.algorithms.als [ 852ms] finished epoch 14 (|ΔP|=115.869, |ΔQ|=0.389)\n",
      "[   INFO] lenskit.algorithms.als [ 908ms] finished epoch 15 (|ΔP|=104.002, |ΔQ|=0.360)\n",
      "[   INFO] lenskit.algorithms.als [ 963ms] finished epoch 16 (|ΔP|=110.334, |ΔQ|=0.357)\n",
      "[   INFO] lenskit.algorithms.als [1.02s] finished epoch 17 (|ΔP|=87.453, |ΔQ|=0.324)\n",
      "[   INFO] lenskit.algorithms.als [1.07s] finished epoch 18 (|ΔP|=99.526, |ΔQ|=0.329)\n",
      "[   INFO] lenskit.algorithms.als [1.13s] finished epoch 19 (|ΔP|=77.154, |ΔQ|=0.301)\n",
      "[   INFO] lenskit.algorithms.als [1.13s] finished training model with 50 features (|P|=2311.787688, |Q|=7.720594)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=50, reg=0.1, w=40) in 1.15s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-mu4rvk48.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.87s\n",
      "[   INFO] lenskit.batch._multi run 35: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-pdzx_jzk.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=50, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.89s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.10s\n",
      "[   INFO] lenskit.batch._multi run 35: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 35: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 36: Popular on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi training algorithm Popular on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm Popular in  12ms\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for Popular\n",
      "[   INFO] lenskit.sharing persisting Popular to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-aqkdsrai.bpk\n",
      "[   INFO] binpickle.write pickled 967 bytes with 7 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with Popular for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in  740ms\n",
      "[   INFO] lenskit.batch._multi generated recommendations in  813ms\n",
      "[   INFO] lenskit.batch._multi run 36: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 36: Popular on ML-Small:4\n",
      "[   INFO] lenskit.batch._multi starting run 37: als.BiasedMF(features=20, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=20, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=20, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9063 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 38ms] normalizing 671x9063 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 46ms] training biased MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als [ 58ms] finished epoch 0 (|ΔP|=32.552, |ΔQ|=123.296)\n",
      "[   INFO] lenskit.algorithms.als [ 71ms] finished epoch 1 (|ΔP|=12.025, |ΔQ|=53.195)\n",
      "[   INFO] lenskit.algorithms.als [ 83ms] finished epoch 2 (|ΔP|=8.976, |ΔQ|=28.077)\n",
      "[   INFO] lenskit.algorithms.als [ 99ms] finished epoch 3 (|ΔP|=6.167, |ΔQ|=17.713)\n",
      "[   INFO] lenskit.algorithms.als [ 115ms] finished epoch 4 (|ΔP|=4.541, |ΔQ|=12.977)\n",
      "[   INFO] lenskit.algorithms.als [ 129ms] finished epoch 5 (|ΔP|=3.523, |ΔQ|=10.003)\n",
      "[   INFO] lenskit.algorithms.als [ 141ms] finished epoch 6 (|ΔP|=2.835, |ΔQ|=8.034)\n",
      "[   INFO] lenskit.algorithms.als [ 154ms] finished epoch 7 (|ΔP|=2.357, |ΔQ|=6.679)\n",
      "[   INFO] lenskit.algorithms.als [ 167ms] finished epoch 8 (|ΔP|=2.012, |ΔQ|=5.693)\n",
      "[   INFO] lenskit.algorithms.als [ 182ms] finished epoch 9 (|ΔP|=1.750, |ΔQ|=4.935)\n",
      "[   INFO] lenskit.algorithms.als [ 196ms] finished epoch 10 (|ΔP|=1.544, |ΔQ|=4.330)\n",
      "[   INFO] lenskit.algorithms.als [ 209ms] finished epoch 11 (|ΔP|=1.379, |ΔQ|=3.843)\n",
      "[   INFO] lenskit.algorithms.als [ 222ms] finished epoch 12 (|ΔP|=1.245, |ΔQ|=3.446)\n",
      "[   INFO] lenskit.algorithms.als [ 235ms] finished epoch 13 (|ΔP|=1.133, |ΔQ|=3.117)\n",
      "[   INFO] lenskit.algorithms.als [ 249ms] finished epoch 14 (|ΔP|=1.039, |ΔQ|=2.840)\n",
      "[   INFO] lenskit.algorithms.als [ 262ms] finished epoch 15 (|ΔP|=0.958, |ΔQ|=2.605)\n",
      "[   INFO] lenskit.algorithms.als [ 276ms] finished epoch 16 (|ΔP|=0.887, |ΔQ|=2.403)\n",
      "[   INFO] lenskit.algorithms.als [ 289ms] finished epoch 17 (|ΔP|=0.825, |ΔQ|=2.228)\n",
      "[   INFO] lenskit.algorithms.als [ 302ms] finished epoch 18 (|ΔP|=0.770, |ΔQ|=2.076)\n",
      "[   INFO] lenskit.algorithms.als [ 314ms] finished epoch 19 (|ΔP|=0.721, |ΔQ|=1.944)\n",
      "[   INFO] lenskit.algorithms.als trained model in  316ms (|P|=29.778578, |Q|=94.413827)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=20, regularization=0.1) in  332ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-lw9xrplj.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.95s\n",
      "[   INFO] lenskit.batch._multi run 37: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=20, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=20, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-mslljrms.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=20, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.91s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.11s\n",
      "[   INFO] lenskit.batch._multi run 37: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 37: als.BiasedMF(features=20, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 38: als.BiasedMF(features=30, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=30, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=30, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9063 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 38ms] normalizing 671x9063 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 48ms] training biased MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als [ 64ms] finished epoch 0 (|ΔP|=33.353, |ΔQ|=127.417)\n",
      "[   INFO] lenskit.algorithms.als [ 80ms] finished epoch 1 (|ΔP|=12.688, |ΔQ|=53.269)\n",
      "[   INFO] lenskit.algorithms.als [ 98ms] finished epoch 2 (|ΔP|=9.149, |ΔQ|=26.745)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 3 (|ΔP|=6.246, |ΔQ|=17.242)\n",
      "[   INFO] lenskit.algorithms.als [ 135ms] finished epoch 4 (|ΔP|=4.520, |ΔQ|=12.615)\n",
      "[   INFO] lenskit.algorithms.als [ 154ms] finished epoch 5 (|ΔP|=3.464, |ΔQ|=9.722)\n",
      "[   INFO] lenskit.algorithms.als [ 173ms] finished epoch 6 (|ΔP|=2.781, |ΔQ|=7.810)\n",
      "[   INFO] lenskit.algorithms.als [ 192ms] finished epoch 7 (|ΔP|=2.307, |ΔQ|=6.468)\n",
      "[   INFO] lenskit.algorithms.als [ 211ms] finished epoch 8 (|ΔP|=1.956, |ΔQ|=5.478)\n",
      "[   INFO] lenskit.algorithms.als [ 229ms] finished epoch 9 (|ΔP|=1.684, |ΔQ|=4.724)\n",
      "[   INFO] lenskit.algorithms.als [ 246ms] finished epoch 10 (|ΔP|=1.467, |ΔQ|=4.134)\n",
      "[   INFO] lenskit.algorithms.als [ 263ms] finished epoch 11 (|ΔP|=1.294, |ΔQ|=3.658)\n",
      "[   INFO] lenskit.algorithms.als [ 282ms] finished epoch 12 (|ΔP|=1.156, |ΔQ|=3.266)\n",
      "[   INFO] lenskit.algorithms.als [ 301ms] finished epoch 13 (|ΔP|=1.043, |ΔQ|=2.941)\n",
      "[   INFO] lenskit.algorithms.als [ 320ms] finished epoch 14 (|ΔP|=0.950, |ΔQ|=2.669)\n",
      "[   INFO] lenskit.algorithms.als [ 339ms] finished epoch 15 (|ΔP|=0.872, |ΔQ|=2.439)\n",
      "[   INFO] lenskit.algorithms.als [ 360ms] finished epoch 16 (|ΔP|=0.804, |ΔQ|=2.241)\n",
      "[   INFO] lenskit.algorithms.als [ 379ms] finished epoch 17 (|ΔP|=0.745, |ΔQ|=2.070)\n",
      "[   INFO] lenskit.algorithms.als [ 398ms] finished epoch 18 (|ΔP|=0.694, |ΔQ|=1.920)\n",
      "[   INFO] lenskit.algorithms.als [ 417ms] finished epoch 19 (|ΔP|=0.649, |ΔQ|=1.788)\n",
      "[   INFO] lenskit.algorithms.als trained model in  418ms (|P|=30.777912, |Q|=94.579922)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=30, regularization=0.1) in  433ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-m9get0dy.bpk\n",
      "[   INFO] binpickle.write pickled 1338 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 38: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=30, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=30, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-a4it8oq3.bpk\n",
      "[   INFO] binpickle.write pickled 1338 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=30, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.91s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.11s\n",
      "[   INFO] lenskit.batch._multi run 38: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 38: als.BiasedMF(features=30, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 39: als.BiasedMF(features=40, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=40, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=40, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9063 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 39ms] normalizing 671x9063 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 51ms] training biased MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als [ 72ms] finished epoch 0 (|ΔP|=34.240, |ΔQ|=128.070)\n",
      "[   INFO] lenskit.algorithms.als [ 94ms] finished epoch 1 (|ΔP|=12.770, |ΔQ|=54.373)\n",
      "[   INFO] lenskit.algorithms.als [ 117ms] finished epoch 2 (|ΔP|=9.374, |ΔQ|=27.261)\n",
      "[   INFO] lenskit.algorithms.als [ 141ms] finished epoch 3 (|ΔP|=6.214, |ΔQ|=17.317)\n",
      "[   INFO] lenskit.algorithms.als [ 163ms] finished epoch 4 (|ΔP|=4.430, |ΔQ|=12.690)\n",
      "[   INFO] lenskit.algorithms.als [ 186ms] finished epoch 5 (|ΔP|=3.361, |ΔQ|=9.755)\n",
      "[   INFO] lenskit.algorithms.als [ 211ms] finished epoch 6 (|ΔP|=2.664, |ΔQ|=7.759)\n",
      "[   INFO] lenskit.algorithms.als [ 235ms] finished epoch 7 (|ΔP|=2.182, |ΔQ|=6.350)\n",
      "[   INFO] lenskit.algorithms.als [ 257ms] finished epoch 8 (|ΔP|=1.832, |ΔQ|=5.325)\n",
      "[   INFO] lenskit.algorithms.als [ 282ms] finished epoch 9 (|ΔP|=1.568, |ΔQ|=4.549)\n",
      "[   INFO] lenskit.algorithms.als [ 307ms] finished epoch 10 (|ΔP|=1.362, |ΔQ|=3.939)\n",
      "[   INFO] lenskit.algorithms.als [ 332ms] finished epoch 11 (|ΔP|=1.197, |ΔQ|=3.448)\n",
      "[   INFO] lenskit.algorithms.als [ 355ms] finished epoch 12 (|ΔP|=1.063, |ΔQ|=3.048)\n",
      "[   INFO] lenskit.algorithms.als [ 379ms] finished epoch 13 (|ΔP|=0.952, |ΔQ|=2.720)\n",
      "[   INFO] lenskit.algorithms.als [ 402ms] finished epoch 14 (|ΔP|=0.859, |ΔQ|=2.445)\n",
      "[   INFO] lenskit.algorithms.als [ 426ms] finished epoch 15 (|ΔP|=0.780, |ΔQ|=2.211)\n",
      "[   INFO] lenskit.algorithms.als [ 450ms] finished epoch 16 (|ΔP|=0.712, |ΔQ|=2.011)\n",
      "[   INFO] lenskit.algorithms.als [ 474ms] finished epoch 17 (|ΔP|=0.654, |ΔQ|=1.838)\n",
      "[   INFO] lenskit.algorithms.als [ 498ms] finished epoch 18 (|ΔP|=0.603, |ΔQ|=1.690)\n",
      "[   INFO] lenskit.algorithms.als [ 522ms] finished epoch 19 (|ΔP|=0.559, |ΔQ|=1.563)\n",
      "[   INFO] lenskit.algorithms.als trained model in  525ms (|P|=31.319241, |Q|=94.628291)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=40, regularization=0.1) in  542ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=40, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-oin4b17q.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 39: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=40, regularization=0.1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=40, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-9a6a1d1_.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=40, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.92s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.12s\n",
      "[   INFO] lenskit.batch._multi run 39: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 39: als.BiasedMF(features=40, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 40: als.BiasedMF(features=50, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.BiasedMF(features=50, regularization=0.1) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.BiasedMF(features=50, regularization=0.1) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 0ms] fitting bias model\n",
      "[   INFO] lenskit.algorithms.basic building bias model for 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic global mean: 3.543\n",
      "[   INFO] lenskit.algorithms.basic computed means for 9063 items\n",
      "[   INFO] lenskit.algorithms.basic computed means for 671 users\n",
      "[   INFO] lenskit.algorithms.als [ 37ms] normalizing 671x9063 matrix (99334 nnz)\n",
      "[   INFO] lenskit.algorithms.als [ 49ms] training biased MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als [ 77ms] finished epoch 0 (|ΔP|=34.147, |ΔQ|=128.724)\n",
      "[   INFO] lenskit.algorithms.als [ 106ms] finished epoch 1 (|ΔP|=13.005, |ΔQ|=54.757)\n",
      "[   INFO] lenskit.algorithms.als [ 134ms] finished epoch 2 (|ΔP|=9.463, |ΔQ|=27.389)\n",
      "[   INFO] lenskit.algorithms.als [ 163ms] finished epoch 3 (|ΔP|=6.203, |ΔQ|=17.411)\n",
      "[   INFO] lenskit.algorithms.als [ 189ms] finished epoch 4 (|ΔP|=4.376, |ΔQ|=12.652)\n",
      "[   INFO] lenskit.algorithms.als [ 218ms] finished epoch 5 (|ΔP|=3.266, |ΔQ|=9.623)\n",
      "[   INFO] lenskit.algorithms.als [ 246ms] finished epoch 6 (|ΔP|=2.551, |ΔQ|=7.618)\n",
      "[   INFO] lenskit.algorithms.als [ 274ms] finished epoch 7 (|ΔP|=2.065, |ΔQ|=6.236)\n",
      "[   INFO] lenskit.algorithms.als [ 304ms] finished epoch 8 (|ΔP|=1.718, |ΔQ|=5.231)\n",
      "[   INFO] lenskit.algorithms.als [ 334ms] finished epoch 9 (|ΔP|=1.458, |ΔQ|=4.463)\n",
      "[   INFO] lenskit.algorithms.als [ 365ms] finished epoch 10 (|ΔP|=1.256, |ΔQ|=3.847)\n",
      "[   INFO] lenskit.algorithms.als [ 397ms] finished epoch 11 (|ΔP|=1.095, |ΔQ|=3.335)\n",
      "[   INFO] lenskit.algorithms.als [ 428ms] finished epoch 12 (|ΔP|=0.964, |ΔQ|=2.908)\n",
      "[   INFO] lenskit.algorithms.als [ 457ms] finished epoch 13 (|ΔP|=0.855, |ΔQ|=2.555)\n",
      "[   INFO] lenskit.algorithms.als [ 487ms] finished epoch 14 (|ΔP|=0.763, |ΔQ|=2.264)\n",
      "[   INFO] lenskit.algorithms.als [ 516ms] finished epoch 15 (|ΔP|=0.686, |ΔQ|=2.020)\n",
      "[   INFO] lenskit.algorithms.als [ 544ms] finished epoch 16 (|ΔP|=0.620, |ΔQ|=1.814)\n",
      "[   INFO] lenskit.algorithms.als [ 570ms] finished epoch 17 (|ΔP|=0.564, |ΔQ|=1.639)\n",
      "[   INFO] lenskit.algorithms.als [ 599ms] finished epoch 18 (|ΔP|=0.515, |ΔQ|=1.491)\n",
      "[   INFO] lenskit.algorithms.als [ 628ms] finished epoch 19 (|ΔP|=0.473, |ΔQ|=1.364)\n",
      "[   INFO] lenskit.algorithms.als trained model in  631ms (|P|=31.554301, |Q|=94.674680)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.BiasedMF(features=50, regularization=0.1) in  648ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-3h1q8sef.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.91s\n",
      "[   INFO] lenskit.batch._multi run 40: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.BiasedMF(features=50, regularization=0.1)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.BiasedMF(features=50, regularization=0.1) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-2nsbi2er.bpk\n",
      "[   INFO] binpickle.write pickled 1339 bytes with 10 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.BiasedMF(features=50, regularization=0.1) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.92s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.13s\n",
      "[   INFO] lenskit.batch._multi run 40: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 40: als.BiasedMF(features=50, regularization=0.1) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 41: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=20, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=20, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 13ms] training implicit MF model with ALS for 20 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9063 items\n",
      "[   INFO] lenskit.algorithms.als [ 58ms] finished epoch 0 (|ΔP|=3355.018, |ΔQ|=8.951)\n",
      "[   INFO] lenskit.algorithms.als [ 106ms] finished epoch 1 (|ΔP|=175.236, |ΔQ|=3.678)\n",
      "[   INFO] lenskit.algorithms.als [ 154ms] finished epoch 2 (|ΔP|=67.529, |ΔQ|=2.475)\n",
      "[   INFO] lenskit.algorithms.als [ 202ms] finished epoch 3 (|ΔP|=60.492, |ΔQ|=1.862)\n",
      "[   INFO] lenskit.algorithms.als [ 251ms] finished epoch 4 (|ΔP|=65.468, |ΔQ|=1.589)\n",
      "[   INFO] lenskit.algorithms.als [ 299ms] finished epoch 5 (|ΔP|=69.249, |ΔQ|=1.380)\n",
      "[   INFO] lenskit.algorithms.als [ 347ms] finished epoch 6 (|ΔP|=70.908, |ΔQ|=1.193)\n",
      "[   INFO] lenskit.algorithms.als [ 395ms] finished epoch 7 (|ΔP|=69.268, |ΔQ|=1.017)\n",
      "[   INFO] lenskit.algorithms.als [ 442ms] finished epoch 8 (|ΔP|=61.565, |ΔQ|=0.900)\n",
      "[   INFO] lenskit.algorithms.als [ 489ms] finished epoch 9 (|ΔP|=62.983, |ΔQ|=0.815)\n",
      "[   INFO] lenskit.algorithms.als [ 537ms] finished epoch 10 (|ΔP|=56.960, |ΔQ|=0.724)\n",
      "[   INFO] lenskit.algorithms.als [ 585ms] finished epoch 11 (|ΔP|=64.989, |ΔQ|=0.700)\n",
      "[   INFO] lenskit.algorithms.als [ 635ms] finished epoch 12 (|ΔP|=53.611, |ΔQ|=0.623)\n",
      "[   INFO] lenskit.algorithms.als [ 683ms] finished epoch 13 (|ΔP|=55.810, |ΔQ|=0.595)\n",
      "[   INFO] lenskit.algorithms.als [ 732ms] finished epoch 14 (|ΔP|=53.119, |ΔQ|=0.543)\n",
      "[   INFO] lenskit.algorithms.als [ 779ms] finished epoch 15 (|ΔP|=66.903, |ΔQ|=0.558)\n",
      "[   INFO] lenskit.algorithms.als [ 827ms] finished epoch 16 (|ΔP|=51.305, |ΔQ|=0.498)\n",
      "[   INFO] lenskit.algorithms.als [ 874ms] finished epoch 17 (|ΔP|=73.329, |ΔQ|=0.542)\n",
      "[   INFO] lenskit.algorithms.als [ 922ms] finished epoch 18 (|ΔP|=41.683, |ΔQ|=0.445)\n",
      "[   INFO] lenskit.algorithms.als [ 969ms] finished epoch 19 (|ΔP|=59.176, |ΔQ|=0.492)\n",
      "[   INFO] lenskit.algorithms.als [ 971ms] finished training model with 20 features (|P|=2642.705467, |Q|=9.916332)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=20, reg=0.1, w=40) in  986ms\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-ls6b4qve.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.93s\n",
      "[   INFO] lenskit.batch._multi run 41: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=20, reg=0.1, w=40)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=20, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-u2z7d478.bpk\n",
      "[   INFO] binpickle.write pickled 1194 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=20, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.73s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 1.92s\n",
      "[   INFO] lenskit.batch._multi run 41: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 41: als.ImplicitMF(features=20, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 42: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=30, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=30, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 14ms] training implicit MF model with ALS for 30 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9063 items\n",
      "[   INFO] lenskit.algorithms.als [ 64ms] finished epoch 0 (|ΔP|=3811.909, |ΔQ|=8.193)\n",
      "[   INFO] lenskit.algorithms.als [ 115ms] finished epoch 1 (|ΔP|=198.052, |ΔQ|=3.263)\n",
      "[   INFO] lenskit.algorithms.als [ 166ms] finished epoch 2 (|ΔP|=109.306, |ΔQ|=2.168)\n",
      "[   INFO] lenskit.algorithms.als [ 218ms] finished epoch 3 (|ΔP|=117.540, |ΔQ|=1.687)\n",
      "[   INFO] lenskit.algorithms.als [ 269ms] finished epoch 4 (|ΔP|=124.541, |ΔQ|=1.354)\n",
      "[   INFO] lenskit.algorithms.als [ 322ms] finished epoch 5 (|ΔP|=124.002, |ΔQ|=1.106)\n",
      "[   INFO] lenskit.algorithms.als [ 374ms] finished epoch 6 (|ΔP|=97.685, |ΔQ|=0.913)\n",
      "[   INFO] lenskit.algorithms.als [ 425ms] finished epoch 7 (|ΔP|=96.469, |ΔQ|=0.812)\n",
      "[   INFO] lenskit.algorithms.als [ 477ms] finished epoch 8 (|ΔP|=89.784, |ΔQ|=0.718)\n",
      "[   INFO] lenskit.algorithms.als [ 529ms] finished epoch 9 (|ΔP|=84.360, |ΔQ|=0.654)\n",
      "[   INFO] lenskit.algorithms.als [ 580ms] finished epoch 10 (|ΔP|=80.133, |ΔQ|=0.595)\n",
      "[   INFO] lenskit.algorithms.als [ 632ms] finished epoch 11 (|ΔP|=78.652, |ΔQ|=0.564)\n",
      "[   INFO] lenskit.algorithms.als [ 685ms] finished epoch 12 (|ΔP|=79.025, |ΔQ|=0.524)\n",
      "[   INFO] lenskit.algorithms.als [ 736ms] finished epoch 13 (|ΔP|=86.098, |ΔQ|=0.520)\n",
      "[   INFO] lenskit.algorithms.als [ 790ms] finished epoch 14 (|ΔP|=66.418, |ΔQ|=0.472)\n",
      "[   INFO] lenskit.algorithms.als [ 842ms] finished epoch 15 (|ΔP|=88.431, |ΔQ|=0.500)\n",
      "[   INFO] lenskit.algorithms.als [ 894ms] finished epoch 16 (|ΔP|=59.289, |ΔQ|=0.423)\n",
      "[   INFO] lenskit.algorithms.als [ 945ms] finished epoch 17 (|ΔP|=80.883, |ΔQ|=0.450)\n",
      "[   INFO] lenskit.algorithms.als [ 997ms] finished epoch 18 (|ΔP|=53.785, |ΔQ|=0.384)\n",
      "[   INFO] lenskit.algorithms.als [1.05s] finished epoch 19 (|ΔP|=93.311, |ΔQ|=0.437)\n",
      "[   INFO] lenskit.algorithms.als [1.05s] finished training model with 30 features (|P|=2702.199180, |Q|=9.126627)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=30, reg=0.1, w=40) in 1.06s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-ztz6xd_w.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.88s\n",
      "[   INFO] lenskit.batch._multi run 42: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=30, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=30, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-z3on2twx.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=30, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.82s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.02s\n",
      "[   INFO] lenskit.batch._multi run 42: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 42: als.ImplicitMF(features=30, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 43: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=40, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=40, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 15ms] training implicit MF model with ALS for 40 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9063 items\n",
      "[   INFO] lenskit.algorithms.als [ 66ms] finished epoch 0 (|ΔP|=4088.491, |ΔQ|=8.081)\n",
      "[   INFO] lenskit.algorithms.als [ 118ms] finished epoch 1 (|ΔP|=204.760, |ΔQ|=3.054)\n",
      "[   INFO] lenskit.algorithms.als [ 170ms] finished epoch 2 (|ΔP|=137.835, |ΔQ|=1.979)\n",
      "[   INFO] lenskit.algorithms.als [ 223ms] finished epoch 3 (|ΔP|=171.879, |ΔQ|=1.517)\n",
      "[   INFO] lenskit.algorithms.als [ 275ms] finished epoch 4 (|ΔP|=159.322, |ΔQ|=1.184)\n",
      "[   INFO] lenskit.algorithms.als [ 328ms] finished epoch 5 (|ΔP|=134.153, |ΔQ|=0.943)\n",
      "[   INFO] lenskit.algorithms.als [ 381ms] finished epoch 6 (|ΔP|=122.469, |ΔQ|=0.800)\n",
      "[   INFO] lenskit.algorithms.als [ 434ms] finished epoch 7 (|ΔP|=110.212, |ΔQ|=0.694)\n",
      "[   INFO] lenskit.algorithms.als [ 487ms] finished epoch 8 (|ΔP|=112.739, |ΔQ|=0.630)\n",
      "[   INFO] lenskit.algorithms.als [ 541ms] finished epoch 9 (|ΔP|=95.057, |ΔQ|=0.557)\n",
      "[   INFO] lenskit.algorithms.als [ 595ms] finished epoch 10 (|ΔP|=99.849, |ΔQ|=0.519)\n",
      "[   INFO] lenskit.algorithms.als [ 649ms] finished epoch 11 (|ΔP|=92.813, |ΔQ|=0.477)\n",
      "[   INFO] lenskit.algorithms.als [ 704ms] finished epoch 12 (|ΔP|=90.571, |ΔQ|=0.451)\n",
      "[   INFO] lenskit.algorithms.als [ 760ms] finished epoch 13 (|ΔP|=82.357, |ΔQ|=0.416)\n",
      "[   INFO] lenskit.algorithms.als [ 827ms] finished epoch 14 (|ΔP|=87.906, |ΔQ|=0.410)\n",
      "[   INFO] lenskit.algorithms.als [ 884ms] finished epoch 15 (|ΔP|=80.260, |ΔQ|=0.375)\n",
      "[   INFO] lenskit.algorithms.als [ 936ms] finished epoch 16 (|ΔP|=79.077, |ΔQ|=0.366)\n",
      "[   INFO] lenskit.algorithms.als [ 990ms] finished epoch 17 (|ΔP|=87.854, |ΔQ|=0.361)\n",
      "[   INFO] lenskit.algorithms.als [1.04s] finished epoch 18 (|ΔP|=74.162, |ΔQ|=0.332)\n",
      "[   INFO] lenskit.algorithms.als [1.10s] finished epoch 19 (|ΔP|=68.013, |ΔQ|=0.316)\n",
      "[   INFO] lenskit.algorithms.als [1.10s] finished training model with 40 features (|P|=2798.864673, |Q|=8.851169)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=40, reg=0.1, w=40) in 1.12s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-2dx388rr.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.89s\n",
      "[   INFO] lenskit.batch._multi run 43: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=40, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=40, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-rn022yb5.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=40, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.86s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.06s\n",
      "[   INFO] lenskit.batch._multi run 43: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 43: als.ImplicitMF(features=40, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 44: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi adapting als.ImplicitMF(features=50, reg=0.1, w=40) into a recommender\n",
      "[   INFO] lenskit.batch._multi training algorithm als.ImplicitMF(features=50, reg=0.1, w=40) on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.als [ 17ms] training implicit MF model with ALS for 50 features\n",
      "[   INFO] lenskit.algorithms.als have 99334 observations for 671 users and 9063 items\n",
      "[   INFO] lenskit.algorithms.als [ 69ms] finished epoch 0 (|ΔP|=4314.240, |ΔQ|=7.350)\n",
      "[   INFO] lenskit.algorithms.als [ 123ms] finished epoch 1 (|ΔP|=243.305, |ΔQ|=2.852)\n",
      "[   INFO] lenskit.algorithms.als [ 181ms] finished epoch 2 (|ΔP|=218.755, |ΔQ|=1.829)\n",
      "[   INFO] lenskit.algorithms.als [ 236ms] finished epoch 3 (|ΔP|=241.981, |ΔQ|=1.347)\n",
      "[   INFO] lenskit.algorithms.als [ 291ms] finished epoch 4 (|ΔP|=213.236, |ΔQ|=1.029)\n",
      "[   INFO] lenskit.algorithms.als [ 346ms] finished epoch 5 (|ΔP|=170.938, |ΔQ|=0.827)\n",
      "[   INFO] lenskit.algorithms.als [ 403ms] finished epoch 6 (|ΔP|=151.324, |ΔQ|=0.705)\n",
      "[   INFO] lenskit.algorithms.als [ 458ms] finished epoch 7 (|ΔP|=146.010, |ΔQ|=0.623)\n",
      "[   INFO] lenskit.algorithms.als [ 513ms] finished epoch 8 (|ΔP|=126.886, |ΔQ|=0.552)\n",
      "[   INFO] lenskit.algorithms.als [ 568ms] finished epoch 9 (|ΔP|=125.140, |ΔQ|=0.508)\n",
      "[   INFO] lenskit.algorithms.als [ 623ms] finished epoch 10 (|ΔP|=109.817, |ΔQ|=0.456)\n",
      "[   INFO] lenskit.algorithms.als [ 682ms] finished epoch 11 (|ΔP|=121.407, |ΔQ|=0.441)\n",
      "[   INFO] lenskit.algorithms.als [ 746ms] finished epoch 12 (|ΔP|=105.041, |ΔQ|=0.405)\n",
      "[   INFO] lenskit.algorithms.als [ 804ms] finished epoch 13 (|ΔP|=108.027, |ΔQ|=0.394)\n",
      "[   INFO] lenskit.algorithms.als [ 858ms] finished epoch 14 (|ΔP|=100.251, |ΔQ|=0.369)\n",
      "[   INFO] lenskit.algorithms.als [ 914ms] finished epoch 15 (|ΔP|=115.007, |ΔQ|=0.365)\n",
      "[   INFO] lenskit.algorithms.als [ 969ms] finished epoch 16 (|ΔP|=97.391, |ΔQ|=0.350)\n",
      "[   INFO] lenskit.algorithms.als [1.02s] finished epoch 17 (|ΔP|=88.497, |ΔQ|=0.329)\n",
      "[   INFO] lenskit.algorithms.als [1.08s] finished epoch 18 (|ΔP|=84.539, |ΔQ|=0.310)\n",
      "[   INFO] lenskit.algorithms.als [1.14s] finished epoch 19 (|ΔP|=91.381, |ΔQ|=0.305)\n",
      "[   INFO] lenskit.algorithms.als [1.14s] finished training model with 50 features (|P|=2596.291220, |Q|=8.136999)\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm als.ImplicitMF(features=50, reg=0.1, w=40) in 1.15s\n",
      "[   INFO] lenskit.batch._multi generating 670 predictions for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-wocg76xo.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._predict generating 670 predictions for 134 users\n",
      "[   INFO] lenskit.batch._multi generated predictions in 1.97s\n",
      "[   INFO] lenskit.batch._multi run 44: writing results to my-eval\\predictions.parquet\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for TopN/als.ImplicitMF(features=50, reg=0.1, w=40)\n",
      "[   INFO] lenskit.sharing persisting TopN/als.ImplicitMF(features=50, reg=0.1, w=40) to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-lkdqyfqa.bpk\n",
      "[   INFO] binpickle.write pickled 1193 bytes with 8 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with TopN/als.ImplicitMF(features=50, reg=0.1, w=40) for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in 1.98s\n",
      "[   INFO] lenskit.batch._multi generated recommendations in 2.17s\n",
      "[   INFO] lenskit.batch._multi run 44: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 44: als.ImplicitMF(features=50, reg=0.1, w=40) on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi starting run 45: Popular on ML-Small:5\n",
      "[   INFO] lenskit.batch._multi training algorithm Popular on 99334 ratings\n",
      "[   INFO] lenskit.algorithms.basic trained unrated candidate selector for 99334 ratings\n",
      "[   INFO] lenskit.batch._multi trained algorithm Popular in  13ms\n",
      "[   INFO] lenskit.batch._multi generating recommendations for 134 users for Popular\n",
      "[   INFO] lenskit.sharing persisting Popular to C:\\Users\\MICHAE~1\\AppData\\Local\\Temp\\lkpy-c79p1keh.bpk\n",
      "[   INFO] binpickle.write pickled 967 bytes with 7 buffers\n",
      "[   INFO] lenskit.util.parallel setting up ProcessPoolExecutor w/ 4 workers\n",
      "[   INFO] lenskit.batch._recommend recommending with Popular for 134 users (n_jobs=None)\n",
      "[   INFO] lenskit.batch._recommend recommended for 134 users in  802ms\n",
      "[   INFO] lenskit.batch._multi generated recommendations in  874ms\n",
      "[   INFO] lenskit.batch._multi run 45: writing results to my-eval\\recommendations.parquet\n",
      "[   INFO] lenskit.batch._multi finished run 45: Popular on ML-Small:5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eval.run(progress=tqdm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "Now that the experiment is run, we can read its outputs.\n",
    "\n",
    "First the run metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DataSet</th>\n",
       "      <th>Partition</th>\n",
       "      <th>AlgoClass</th>\n",
       "      <th>AlgoStr</th>\n",
       "      <th>name</th>\n",
       "      <th>features</th>\n",
       "      <th>TrainTime</th>\n",
       "      <th>PredTime</th>\n",
       "      <th>RecTime</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RunId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ML-Small</td>\n",
       "      <td>1</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>als.BiasedMF(features=20, regularization=0.1)</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>20.0</td>\n",
       "      <td>5.426783</td>\n",
       "      <td>2.066797</td>\n",
       "      <td>2.140798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ML-Small</td>\n",
       "      <td>1</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>als.BiasedMF(features=30, regularization=0.1)</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.444219</td>\n",
       "      <td>2.011823</td>\n",
       "      <td>2.139281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ML-Small</td>\n",
       "      <td>1</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>als.BiasedMF(features=40, regularization=0.1)</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.496614</td>\n",
       "      <td>1.942132</td>\n",
       "      <td>2.097481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ML-Small</td>\n",
       "      <td>1</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>als.BiasedMF(features=50, regularization=0.1)</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.633261</td>\n",
       "      <td>1.894300</td>\n",
       "      <td>2.060676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ML-Small</td>\n",
       "      <td>1</td>\n",
       "      <td>ImplicitMF</td>\n",
       "      <td>als.ImplicitMF(features=20, reg=0.1, w=40)</td>\n",
       "      <td>ImplicitMF</td>\n",
       "      <td>20.0</td>\n",
       "      <td>8.072543</td>\n",
       "      <td>2.431842</td>\n",
       "      <td>2.097856</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        DataSet  Partition   AlgoClass  \\\n",
       "RunId                                    \n",
       "1      ML-Small          1    BiasedMF   \n",
       "2      ML-Small          1    BiasedMF   \n",
       "3      ML-Small          1    BiasedMF   \n",
       "4      ML-Small          1    BiasedMF   \n",
       "5      ML-Small          1  ImplicitMF   \n",
       "\n",
       "                                             AlgoStr        name  features  \\\n",
       "RunId                                                                        \n",
       "1      als.BiasedMF(features=20, regularization=0.1)    BiasedMF      20.0   \n",
       "2      als.BiasedMF(features=30, regularization=0.1)    BiasedMF      30.0   \n",
       "3      als.BiasedMF(features=40, regularization=0.1)    BiasedMF      40.0   \n",
       "4      als.BiasedMF(features=50, regularization=0.1)    BiasedMF      50.0   \n",
       "5         als.ImplicitMF(features=20, reg=0.1, w=40)  ImplicitMF      20.0   \n",
       "\n",
       "       TrainTime  PredTime   RecTime  \n",
       "RunId                                 \n",
       "1       5.426783  2.066797  2.140798  \n",
       "2       0.444219  2.011823  2.139281  \n",
       "3       0.496614  1.942132  2.097481  \n",
       "4       0.633261  1.894300  2.060676  \n",
       "5       8.072543  2.431842  2.097856  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "runs = pd.read_csv('my-eval/runs.csv')\n",
    "runs.set_index('RunId', inplace=True)\n",
    "runs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This describes each run - a data set, partition, and algorithm combination.  To evaluate, we need to get the actual recommendations, and combine them with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>score</th>\n",
       "      <th>user</th>\n",
       "      <th>rank</th>\n",
       "      <th>RunId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3469</td>\n",
       "      <td>4.753794</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>913</td>\n",
       "      <td>4.718944</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2064</td>\n",
       "      <td>4.712152</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>969</td>\n",
       "      <td>4.706261</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3088</td>\n",
       "      <td>4.683724</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item     score  user  rank  RunId\n",
       "0  3469  4.753794    10     1      1\n",
       "1   913  4.718944    10     2      1\n",
       "2  2064  4.712152    10     3      1\n",
       "3   969  4.706261    10     4      1\n",
       "4  3088  4.683724    10     5      1"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recs = pd.read_parquet('my-eval/recommendations.parquet')\n",
    "recs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to compute per-(run,user) evaluations of the recommendations *before* combining with metadata. \n",
    "\n",
    "In order to evaluate the recommendation list, we need to build a combined set of truth data. Since this is a disjoint partition of users over a single data set, we can just concatenate the individual test frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "truth = pd.concat((p.test for p in pairs), ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can set up an analysis and compute the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 120780 recommendations (3355 truth rows)\n",
      "[   INFO] lenskit.topn using group columns ['user', 'RunId']\n",
      "[   INFO] lenskit.topn ungrouped columns: ['item', 'score', 'rank']\n",
      "[   INFO] lenskit.topn using truth ID columns ['user']\n",
      "[   INFO] lenskit.topn computing analysis for 6039 lists\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff2edd6e42654a169dbebd01ca1ac7be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=6039.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[   INFO] lenskit.topn analyzed 6039 lists in 2.71s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>ndcg</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user</th>\n",
       "      <th>RunId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">1</th>\n",
       "      <th>28</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.164037</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            nrecs      ndcg\n",
       "user RunId                 \n",
       "1    28      20.0  0.000000\n",
       "     29      20.0  0.000000\n",
       "     30      20.0  0.000000\n",
       "     31      20.0  0.000000\n",
       "     32      20.0  0.164037"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "rla.add_metric(topn.ndcg)\n",
    "raw_ndcg = rla.compute(recs, truth)\n",
    "raw_ndcg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to combine this with our run data, so that we know what algorithms and configurations we are evaluating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>AlgoClass</th>\n",
       "      <th>features</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user</th>\n",
       "      <th>RunId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">1</th>\n",
       "      <th>28</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>40.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>BiasedMF</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>20.0</td>\n",
       "      <td>0.164037</td>\n",
       "      <td>ImplicitMF</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            nrecs      ndcg   AlgoClass  features\n",
       "user RunId                                       \n",
       "1    28      20.0  0.000000    BiasedMF      20.0\n",
       "     29      20.0  0.000000    BiasedMF      30.0\n",
       "     30      20.0  0.000000    BiasedMF      40.0\n",
       "     31      20.0  0.000000    BiasedMF      50.0\n",
       "     32      20.0  0.164037  ImplicitMF      20.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ndcg = raw_ndcg.join(runs[['AlgoClass', 'features']], on='RunId')\n",
    "ndcg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the overall average performance for each algorithm configuration - fillna makes the group-by happy with Popular's lack of a feature count:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AlgoClass   features\n",
       "BiasedMF    20.0        0.016942\n",
       "            30.0        0.018656\n",
       "            40.0        0.022978\n",
       "            50.0        0.024254\n",
       "ImplicitMF  20.0        0.086721\n",
       "            30.0        0.101501\n",
       "            40.0        0.103832\n",
       "            50.0        0.117933\n",
       "Popular     0.0         0.086276\n",
       "Name: ndcg, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ndcg.fillna(0).groupby(['AlgoClass', 'features'])['ndcg'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can plot this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'nDCG')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxV5b3v8c8vM4Q54ESgzKIyBAxQlFFQ8VZxKOLUKqdWO9H2np7q0WtPq15Pq7V1aI9tL1dbsAfnSi/aVqpMakUlKLagWMMgRBAZFIQYMv3uH2vtZCdZIQlkZyfh+3698srea9rPYofnu5/n2etZ5u6IiIjUlpLsAoiISOukgBARkUgKCBERiaSAEBGRSAoIERGJlJbsAjSXnj17er9+/ZJdDBGRNmXNmjW73b1X1Lp2ExD9+vWjoKAg2cUQEWlTzOz9+tapi0lERCIlNCDMbIaZvWtmhWZ2U8T6SWb2hpmVm9msuOV5ZrbKzNab2d/N7LJEllNEROpKWECYWSrwAHAecCpwhZmdWmuzrcAc4JFay4uBq939NGAGcJ+ZdUtUWUVEpK5EjkGMBQrdfROAmT0GXAi8HdvA3beE6yrjd3T3f8Y93m5mHwG9gE8SWF4REYmTyC6m3sC2uOdF4bImMbOxQAawMWLd9WZWYGYFu3btOuKCiohIXYkMCItY1qSZAc3sROD3wL+4e2Xt9e4+z93z3T2/V6/Ib2mJiMgRSmRAFAF94p7nAtsbu7OZdQH+BPzA3V9t5rKJiEgDEjkGsRoYbGb9gQ+Ay4ErG7OjmWUAi4CH3f3JxBVRRKQN+2QrbFwOXgn5/9Lsh09YQLh7uZnNBZYAqcBv3X29md0OFLj7YjMbQxAE3YELzOy28JtLs4FJQI6ZzQkPOcfd1yaqvCIirV7JftjyMmxaDhuXwZ7CYHnumIQEhLWXGwbl5+e7rqQWkXalsgK2vxmEwcblUPQ6VJZDekfoNwEGngUDpkKvk8Gihn0bZmZr3D0/al27mWpDRKRd+HhLEAYbl8HmlVCyDzA4cSSc8Z0gFPqMhbTMhBdFASEikkwl+2DzS9XdRns3Bcu75MIpFwSB0H8KZOe0eNEUECIiLamiHLa/EddttBq8AtKzof9EGPf1oNuo5+Aj7jZqLgoIEZFE27s5DIRlQWvhUNhtdNIomPCvQSshdwykZSS7pDUoIEREmttnn8CWl6pD4eMtwfKufeC0i2DgVOg/GTr2SGoxG6KAEBE5WhVl8MGa6m6jDwqCaxMyOkH/STB+btBtlDMw6d1GTaGAEBFpKvdgMDkWCFtegkP7wVLgpNEw8ftht1E+pKYnu7RHTAEhItIYn30Mm1+s7jb6ZGuwvFtfGHZJ+G2jSdChe3LL2YwUECIiUSrKgm8Yxa5J2P5G0G2U2SUIgtg1CT0GtKluo6ZQQIiIQNBttGdjdQthy8tQ+mnQbdQ7HybdGARC79Mh9dioOo+NsxQRiVK8N7haOTaWsC+8hU33/jDi0iAQ+k2EDsfmDS0VECJy7CgvDeYzigXC9jcBh8yuMGBSeE3C1KDbSBQQItKOucPu92p2G5UdBEsNLkybcnMQCCeNPma6jZpC/yIi0r4c3AObV1S3EvZ/ECzvMRDyrgi7jSZAVtekFrMtUECISNtWfgi2vVYdCDveAjwIgAFTYMANQSuhe7/klrMNUkCISNviDrvere42ev9vUFYMKWmQOxam3hJ2G42ClNRkl7ZNU0CISOt3cDdsWlEdCp/uCJbnDIJRX6ruNsrsnNRitjcKCBFpfcpKYNur1Repffj3YHmH7mG30dSgldCtbzJL2e4pIEQk+dzho3eCMNi0HLb8Dco/g5R06DMOzvqPIBBOzFO3UQtSQIhIchz4KK7baDkc+DBY3nMInH5N0G30uTMhs1NSi3ksU0CISMso+wy2rgq7jZbDzn8Eyzv0CLqNBp4VtBK65iazlBJHASEiieEOO9dXdxu9/wqUlwTdRn0/D9N+FATCCSMhJSXZpZUICggRaT6ffljdbbRpBRzYGSzvNRTyvxJ2G50BGdnJLKU0kgJCRI5caTFsfaW62+ij9cHyjj1rdht1OSmZpZQjpIAQkcarrISd66qvR9j6KlQcgtQM6Dsept8ahMLxw9Vt1A4oIETas8qK4Crj0oPVP2XFUHog+PRfejCYvK70YMQ2Eft89jGU7AuOfdypMPa6oIXQ9wzI6Jjcc5Vmp4AQaQ0qyg5fMccq89IDTajwi4NrCZoivWPwk5Fd/ZPeEbJ7BQGQ2TmYzmLAFOhyYiL+JaQVUUCINJY7VJQewSfx+G3q2aeitAkFMcjoFFTYGdmQHlbkWV2CSjv2PKNjsF1UhR+1f3pHdQtJDQoIaX/cg69TNucn8djzyvLGl8NSw4o4u2Zl3LEndKun8o6vsOur4NOy2u09kKV1UUBI61NZGcy9s+vdsMJuZOUde152MLi5fGOlZkZXxp1PbPyn76j9UzNUkUubltCAMLMZwP1AKvCgu99Za/0k4D5gBHC5uz8Vt+4a4Afh0zvcfUEiyypJVrw3+FZM4QtQuBQOflR3m7QO0ZVxxx6Nq7zr20Z3EhOJlLD/GWaWCjwAnA0UAavNbLG7vx232VZgDvD9Wvv2AH4E5AMOrAn3/ThR5ZUWVlkJO9YGgfDe8/BBQfCpv0N3GDgNBp8NvfODQdGMcOBUk7SJtKhEfnQaCxS6+yYAM3sMuBCoCgh33xKuq90fcC7wvLvvDdc/D8wAHk1geSXRYq2E956HjUvh4C7Aghu7TLoBBp0NvUcrCERaiUQGRG9gW9zzImDcUezbu/ZGZnY9cD1A376aF77VqayEHW/Cey9A4fPwwZqwldADBk0LAmHQNMjumeySikiERAZE1OicN+e+7j4PmAeQn5/f2GNLIhXvDcYQCp8PfhfvBixoGUy6Meg60q0gRdqERAZEEdAn7nkusL0J+06pte+KZimVNK/KStj+ZhAI74WtBBw65lSPJQw8S60EkTYokQGxGhhsZv2BD4DLgSsbue8S4Mdm1j18fg5wc/MXUY7IwT3BGEJsLKF4D0Er4XSYclPQdXSS7vwl0tYlLCDcvdzM5hJU9qnAb919vZndDhS4+2IzGwMsAroDF5jZbe5+mrvvNbP/TRAyALfHBqwlCSorglbCe8+HYwlvELQSesKg6UEgDDwLsnOSXVIRaUbm3j667vPz872goCDZxWg/Du6uOZbw2V7AIDc/CITB0+HEUZqaQaSNM7M17p4ftU5XCEmgsiJoGcTGEra/SVUrYfA51WMJHXsku6Qi0kIUEMeyA7vixhKWBa0ESwkuUJv6v4LuoxPz1EoQOUYpII4llRXBt4xiYwnb1wIeTOU85NwgENRKEJGQAqK9O/BR9VjCxmXBDV8sBXLHwNRbgrEE3TReRCIoINqbygooKqgeS9ixNliefRwMOS8IhAFT1UoQkQYpINqDAx9VT3q3cRmUfBK2EsbCWT8IvnV0wgi1EkSkSRQQbVFFeTD7aWwsYcdbwfJOx8PQL4RjCVODmVFFRI6QAqKt+HRneK+EWCthX3DHsj5j4az/CL6GevxwtRJEpNkoIFqrinIoWl09lvDh34PlnU6AoRdUjyV06JbccopIu6WAaE0+/bB6LGHT8rhWwjiY9sNwLGG4bmMpIi1CAZFMFeVQ9Hr1WMKH/wiWdz4RTrkgCIQBU9RKEJGkUEC0tP074sYSVsChsJXQ9/Mw7UfhWMIwtRJEJOkUEIlWUQbbXg/HEl6AnbFWwklw6swgEAZMgayuySyliEgdCohE2L89bixhBRzaDylp0OfzMP3WoOvo+NPUShCRVk0B0RwqymDba+FYwguwc12wvPNJcNpF1WMJWV2SWUoRkSZRQBypfR9UjyVsWlndSug7HqbfFkyRfdwpaiWISJulgGisijLY+mr1WMJH64PlXXrDaRcHYwn9J6uVICLthgLicPZ9UH2h2qaVUPoppKQH3zg6+/ag60itBBFppxQQ8cpLYdur1WMJH70dLO+SC8O/GI4lTIbMzsktp4hIC1BAlOyHdX8IAmHTCig9ELQSPncGnHNHEAq9TlYrQUSOOe0mIPbs2cP8+fNrLDvttNMYM2YMZWVlLFy4sM4+eXl55A3ugz/7rxxM7U5Rx5F80O0UdmQNodwzye+Sz7DjhrJv3z4WLVpUZ//x48dz8skns3v3bp599tk66ydNmsSAAQP48MMPee655+qsnzZtGn369GHbtm0sXbq0zvoZM2ZwwgknsGnTJl588cU6688//3x69uzJu+++y6pVq+qsv/jii+natSvr1q2joKCgzvrZs2fTsWNH1q5dy9q1a+usv+qqq0hPT2f16tWsX7++zvo5c+YA8Morr/DPf/6zxrr09HSuuuoqAFauXMnmzZtrrO/YsSOzZ88G4IUXXqCoqKjG+i5dunDJJZcA8Nxzz/Hhhx/WWJ+Tk8MFF1wAwDPPPMOePXtqrD/hhBOYMWMGAE8//TT79++vsT43N5fp06cD8MQTT1BcXFxjff/+/Zk8eTIACxcupKysrMb6IUOGcMYZZwDU+buDRv7t5eVRXFzME088UWd9fn4+w4YN09+e/vZa/G8vXrsJiCOWnUPJ9a/y1JJVaiWIiMQxd092GZpFfn6+R31SERGR+pnZGnfPj1qnmweIiEgkBYSIiERSQIiISCQFhIiIRFJAiIhIJAWEiIhEUkCIiEikhAaEmc0ws3fNrNDMbopYn2lmj4frXzOzfuHydDNbYGb/MLN3zOzmRJZTRETqSlhAmFkq8ABwHnAqcIWZnVprs2uBj919EHAvcFe4/FIg092HA6cDX4uFh4iItIxEtiDGAoXuvsndS4HHgAtrbXMhsCB8/BQwzcwMcCDbzNKADkApsB8REWkxiQyI3sC2uOdF4bLIbdy9HNgH5BCExUFgB7AV+Jm77639AmZ2vZkVmFnBrl27mv8MRESOYYmcrC9q5rvaEz/Vt81YoAI4CegOvGRmL7j7phobus8D5kEwF9NRl1hEkq6srIyioiJKSkqSXZR2JSsri9zcXNLT0xu9TyIDogjoE/c8F9hezzZFYXdSV2AvcCXwnLuXAR+Z2d+AfGATItKuFRUV0blzZ/r164dphuVm4e7s2bOHoqIi+vfv3+j9EtnFtBoYbGb9zSwDuBxYXGubxcA14eNZwDIPppfdCpxlgWzg88CGBJZVRFqJkpIScnJyFA7NyMzIyclpcqssYQERjinMBZYA7wBPuPt6M7vdzGaGmz0E5JhZIfA9IPZV2AeATsA6gqD5nbv/PVFlFZHWReHQ/I7k3zShNwxy9z8Df6617Idxj0sIvtJae78DUctFRKTl6EpqEZFaUlNTycvLY9iwYVx66aV1bgt6tObPn8/cuXOb9ZiJoIAQEamlQ4cOrF27lnXr1pGRkcFvfvObpJanvLw8Ka+rgBAROYyJEydSWFgIwD333MOwYcMYNmwY9913HwBbtmxh6NChXHPNNYwYMYJZs2ZVtTj69evH7t27ASgoKGDKlCl1jv/MM88wbtw4Ro0axfTp09m5cycAt956K9dffz3nnHMOV199dQucaV0JHYMQETla8+fPr7PstNNOY8yYMZSVlbFw4cI66/Py8sjLy6O4uJgnnniixro5c+Y0+rXLy8v5y1/+wowZM1izZg2/+93veO2113B3xo0bx+TJk+nevTvvvvsuDz30EGeeeSZf+cpX+NWvfsX3v//9Rr3GhAkTePXVVzEzHnzwQX7605/y85//HIA1a9bw8ssv06FDh0aXuTmpBSEiUstnn31GXl4e+fn59O3bl2uvvZaXX36Ziy++mOzsbDp16sQll1zCSy+9BECfPn0488wzAfjSl77Eyy+/3OjXKioq4txzz2X48OHcfffdrF+/vmrdzJkzkxYOoBaEiLRyh/vEn56eftj1HTt2bFKLISY2BhEvuEQrWu2vkMaep6WlUVlZCVDvNQjf/va3+d73vsfMmTNZsWIFt956a9W67OzsJpe9OakFISLSCJMmTeKPf/wjxcXFHDx4kEWLFjFx4kQAtm7dyqpVqwB49NFHmTBhAhCMQaxZswaAP/zhD5HH3bdvH717B9PULViwIHKbZFFAiIg0wujRo5kzZw5jx45l3LhxfPWrX2XUqFEAnHLKKSxYsIARI0awd+9evvGNbwDwox/9iO9+97tMnDiR1NTUyOPeeuutXHrppUycOJGePXu22Pk0hjXQbLoQyHX3B8LnrwG9wtU3uvtTiS9i4+Tn53tBQUGyiyEiR+mdd97hlFNOSXYxGm3Lli2cf/75rFu3LtlFaVDUv62ZrXH3/KjtG2pB3EjN+ZMygTHAFOAbR15MERFp7RoapM5w9/h7Orzs7nuAPeEkeiIix7R+/fq1idbDkWioBdE9/om7x18b3gsREWm3GgqI18zsutoLzexrwOuJKZKIiLQGDXUx/SvwRzO7EngjXHY6wVjERYksmIiIJNdhA8LdPwLOMLOzgNPCxX9y92UJL5mIiCTVYbuYzGyMmZ3n7svc/ZfhzzIzu8DMTm+pQoqItKTYdN8jR45k9OjRvPLKKwBs376dWbNmJex1V6xYwfnnnw8Ec1CZGUuXLq1av2jRIsyMp54KrjCYMmUKJ598ctXcU7HlzaWhLqa7gTkRy98B5gFnNWtpRERagfipNpYsWcLNN9/MypUrOemkk5q9Ej6c4cOH8+ijjzJt2jQAHnvsMUaOHFljm4ULF5KfH3kZw1FraJA6x9231F7o7oVATkJKJCLSiuzfv5/u3YMvdG7ZsoVhw4ZVPZ44cSKjR4+u0crYsWMHkyZNqrrhUGxCv7/+9a+MHz+e0aNHc+mll3LgwAEAnnvuOYYOHcqECRN4+umna7z2xIkTef311ykrK+PAgQMUFhaSl5fXUqfeYAvicNMI6joIEUmo255Zz9vb9zfrMU89qQs/uuC0w24Tm821pKSEHTt2sGxZ3WHX4447jueff56srCzee+89rrjiCgoKCnjkkUc499xzueWWW6ioqKC4uJjdu3dzxx138MILL5Cdnc1dd93FPffcw4033sh1113HsmXLGDRoEJdddlmN1zAzpk+fzpIlS9i3bx8zZ85k8+bNNba56qqrqmZ8Xbp0KTk5zffZvaGAeMHM/hP4gcfNyWFmtwEaqBaRdim+i2nVqlVcffXVdS6GKysrY+7cuaxdu5bU1FT++c9/AjBmzBi+8pWvUFZWxkUXXUReXh4rV67k7bffrpoSvLS0lPHjx7Nhwwb69+/P4MGDgWCq8Hnz5tV4ncsvv5xf/OIX7Nu3j5///Of8+Mc/rrE+kV1MDQXEvwEPAoVmFpv7Ng9YDXw1ISUSEQk19Em/JYwfP57du3eza9euGsvvvfdejj/+eN566y0qKyvJysoCgllfX3zxRf70pz/x5S9/mRtuuIHu3btz9tln8+ijj9Y4xtq1a+tMFV7b2LFjWbduHR06dGDIkCHNe3INaOhrrgeBK8xsANVfc13v7psSXjIRkVZgw4YNVFRUkJOTU3UrUQim6c7NzSUlJYUFCxZQUVEBwPvvv0/v3r257rrrOHjwIG+88Qa33HIL3/rWtygsLGTQoEEUFxdTVFTE0KFD2bx5Mxs3bmTgwIF1AiTmJz/5SVUAtaQGbxhkZmkE4TA0XORmttXdk3MXbRGRBIuNQUBwo6AFCxbUma77m9/8Jl/84hd58sknmTp1atXNfVasWMHdd99Neno6nTp14uGHH6ZXr17Mnz+fK664gkOHDgFwxx13MGTIEObNm8cXvvAFevbsyYQJEyLndTrvvPMSfMbRGpru+yRgObADeBMwYBRwAjDV3be3RCEbQ9N9i7QPbW2677akqdN9N9SC+DHwa3e/r9YBvwP8BLjmKMoqIiKtWEMB8Xl3n1N7obv/wszeTUyRRESkNWjoQrnPDrOu+DDrRESkjWuoBdHVzC6JWG5AlwSUR0REWomGAmIlcEHc89iItoXrRESknWroOoh/ATCzLOCLQL+4fer/+pOIiLR5DY1BxPyRoCVRBhwIfw42tJOZzTCzd82s0MxuilifaWaPh+tfM7N+cetGmNkqM1tvZv8IQ0pEJOE6derULMeJn7578eLF3HnnnYfd/owzzgCCiQAfeeSRGscxMx566KGqZW+++SZmxs9+9jMA5syZQ//+/aum/v7FL35x1OVv8EK5UK67z2jKgc0sFXgAOBsoAlab2WJ3fztus2uBj919kJldDtwFXBZenPffwJfd/S0zyyEIJxGRNmnmzJnMnDnzsNvEZoSNBcSVV15ZtW748OE8/vjjXHvttUD01N933313s96vorEtiFfMbHgTjz0WKHT3Te5eCjwGXFhrmwuBBeHjp4BpFkxMcg7wd3d/C8Dd97h7RRNfX0TkqKxYsYLJkycze/ZshgwZwk033cTChQsZO3Ysw4cPZ+PGjUDw6f3rX/86EydOZMiQITz77LN1jjV//nzmzp0LwM6dO7n44osZOXIkI0eOrAqGWMvlpptu4qWXXiIvL497770XgL59+1JSUsLOnTtxd5577rmEX2Hd2BbEBGCOmW0GDhEMUru7jzjMPr2BbXHPi4Bx9W3j7uVmto/gPhNDCKb0WAL0Ah5z9582sqwi0l785Sb48B/Ne8wThsN5h+/qiffWW2/xzjvv0KNHDwYMGMBXv/pVXn/9de6//35++ctfct99wXXEW7ZsYeXKlWzcuJGpU6dSWFhY7zG/853vMHnyZBYtWkRFRUXVvSFi7rzzTn72s59VBc2KFSsAmDVrFk8++SSjRo1i9OjRZGZm1tjvhhtu4I477gDg97//PcOHN/VzfU2NDYgjiamoKQprD2zXt00aQSiNIbjeYml4OfjSGjubXQ9cD0G6iog0tzFjxnDiiScCMHDgQM455xwg6PJZvnx51XazZ88mJSWFwYMHM2DAADZs2FDvMZctW8bDDz8MBLc37dq1a6PKMnv2bC677DI2bNjAFVdcUdXyiGnuLqZGBYS7v38Exy4C+sQ9zwVqz90U26YoHHfoCuwNl690990AZvZnYDRQIyDcfR7BrU/Jz8/Xt6pE2psmfNJPlPhP6SkpKVXPU1JSKC+vnrO09rTdDU3jfSROOOEE0tPTef7557n//vvrBERza+wYxJFYDQw2s/5mlgFcDiyutc1iqudzmgUsC29MtAQYYWYdw+CYDLyNiEgr9eSTT1JZWcnGjRvZtGkTJ598cr3bTps2jV//+tcAVFRUsH9/zbvmde7cmU8//TRy39tvv5277rqrzuyyiZCwgAinA59LUNm/Azzh7uvN7HYziw3lPwTkmFkh8D3gpnDfj4F7CEJmLfCGu/8pUWUVETlaJ598MpMnT+a8887jN7/5zWHv33D//fezfPlyhg8fzumnn8769etrrB8xYgRpaWmMHDmyapA65owzzuCiiy5KyDnUdtjpvtsSTfct0j60xem+58yZw/nnn9+s/f+J0NTpvhPZxSQiIm1YY7/FJCIi9Zg/f36yi5AQakGISKvTXrq+W5Mj+TdVQIhIq5KVlcWePXsUEs3I3dmzZ89hB86jqItJRFqV3NxcioqK2LVrV7KL0q5kZWWRm5vbpH0UECLSqqSnp9O/f/9kF0NQF5OIiNRDASEiIpEUECIiEkkBISIikRQQIiISSQEhIiKRFBAiIhJJASEiIpEUECIiEkkBISIikRQQIiISSQEhIiKRFBAiIhJJASEiIpEUECIiEkkBISIikRQQIiISSQEhIiKRFBAiIhJJASEiIpEUECIiEkkBISIikRQQIiISSQEhIiKRFBAiIhIpoQFhZjPM7F0zKzSzmyLWZ5rZ4+H618ysX631fc3sgJl9P5HlFBGRuhIWEGaWCjwAnAecClxhZqfW2uxa4GN3HwTcC9xVa/29wF8SVUYREalfIlsQY4FCd9/k7qXAY8CFtba5EFgQPn4KmGZmBmBmFwGbgPUJLKOIiNQjkQHRG9gW97woXBa5jbuXA/uAHDPLBv4duO1wL2Bm15tZgZkV7Nq1q9kKLiIiiQ0Ii1jmjdzmNuBedz9wuBdw93nunu/u+b169TrCYoqISJS0BB67COgT9zwX2F7PNkVmlgZ0BfYC44BZZvZToBtQaWYl7v5fCSyviIjESWRArAYGm1l/4APgcuDKWtssBq4BVgGzgGXu7sDE2AZmditwQOEgItKyEhYQ7l5uZnOBJUAq8Ft3X29mtwMF7r4YeAj4vZkVErQcLk9UeUREpGks+MDe9uXn53tBQUGyiyEi0qaY2Rp3z49apyupRUQkkgJCREQiKSBERCSSAkJERCIpIEREJJICQkREIikgREQkkgJCREQiKSBERCSSAkJERCIpIEREJJICQkREIikgREQkkgJCREQiKSBERCSSAkJERCIpIEREJJICQkREIikgREQkkgJCREQiKSBERCSSAkJERCIpIEREJFJasgsgIiKNU1HplJZXUlpRSVn4U1peSWqKkdu9Y7O/ngJCRASorHTKKoMKt6zCqyrfqsq43CmtqKC0vHpdWUVsvVc9L6uo5FB5zQq8rMKD7eKOV1pevSx2nOpjeI3jxx5XenTZR/XtxqJvntns/yYKCBFJOHenvLJ2petVFWZVZRhXOcdXstX71a2cqyrVsOI+VKvSLQsr4tIax/M6lXh5fbXvUUhLMdJTU0hPNTLSUslINdLTUkhPTSEjNYX0tBQyUo3M9BQ6ZaUFy9PCdalW63nwOHa8zKrHKfTsnNnsZQcFhIjEcXf2f1bOnoOH2HuwlN0HStl7sJS9Bw+x+0ApnxSXUlLW8CfeOp+cKyrx5q9/yUhLITOsaINKuLryja9Mu2SkB5VzrYo2I26fOpVzWs3jZMS9TnzlnJ6aEve85vFSU6z5T7oFKSBE2rGGKvzgcSm7DwTrPy4upawiuibvlJlG9+x0OqSn1qhkO2akHaZyjlW6YeWcVvOTc3TlXF3R1v7kXH1sIzXFMGvbFXBrp4AQaUPcnf0l5ewJK/Q9B0vZcyCo8Ksfx5YfarDCz+mUQY/sDHK7d2Bkbjd6dMogJzsjXJ5Z9bh7xwyy0lNb+Gwl2RQQIkkUq/D3hhV6c1X4vRVTLLIAAAmLSURBVLtlMaJ3V1X4clQSGhBmNgO4H0gFHnT3O2utzwQeBk4H9gCXufsWMzsbuBPIAEqBG9x9WSLLKtIcoir8uo9Lw8dBK+BwFX6PsELv3S2L4b27kNMpqOSD5dWPe2Srwpfml7CAMLNU4AHgbKAIWG1mi9397bjNrgU+dvdBZnY5cBdwGbAbuMDdt5vZMGAJ0DtRZRWpT3yFX1+/vSp8aa8S2YIYCxS6+yYAM3sMuBCID4gLgVvDx08B/2Vm5u5vxm2zHsgys0x3P5TA8soxoDEVfvVgrip8ObYlMiB6A9vinhcB4+rbxt3LzWwfkEPQgoj5IvBmVDiY2fXA9QB9+/ZtvpJLm+HufHqovLrfvurTfOyT/SFV+CJHKJEBEfX9s9r/Mw+7jZmdRtDtdE7UC7j7PGAeQH5+fgK+ZS0tobIyuGjpUHkFJWXVvw8cKqu3wo9/3FCFHxu0Hd67Cz2yM+nZqbqS79kpUxW+SD0SGRBFQJ+457nA9nq2KTKzNKArsBfAzHKBRcDV7r4xgeWUUHl49WpJWUVYYVc/ru/3oYjn8ZV8/b+r9y2tqGxU+eIr/JO6ZTFMFb5IQiUyIFYDg82sP/ABcDlwZa1tFgPXAKuAWcAyd3cz6wb8CbjZ3f+WwDK2Ou4eTgNQXaE2pmIuKavgUFklJeU1f9dXuZdGLD+aqQbMICstlcz0lDq/M9NSyM5Mo0d2CpnpqWSmpZDViN8dM1JV4YskUcICIhxTmEvwDaRU4Lfuvt7MbgcK3H0x8BDwezMrJGg5XB7uPhcYBPyHmf1HuOwcd/8oUeWNKH/Vp+g6n5Lr+1RcVkFJeWVkRV1dYVf/ru8T9tFMCZOaYmSl1a2IY8+7dEinV+fMuIo4hcy01Hp/Rx2jRmUeBkF6qq5qFWlvzBMxQUoS5Ofne0FBQZP323uwlMv+z6rIT9pHI5ivJa6irVHx1v/JObKSPlwlnp5aFQhZaSmkpeoWHyLSeGa2xt3zo9Yd81dSZ6alMOi4TtGVdWO6Q2p0pVRX3m19ki4RkWM+ILIz0/j1l05PdjFERFod9UeIiEgkBYSIiERSQIiISCQFhIiIRFJAiIhIJAWEiIhEUkCIiEgkBYSIiERqN1NtmNku4P2jOERPat6Hoq1qL+cBOpfWqr2cS3s5Dzi6c/mcu/eKWtFuAuJomVlBffORtCXt5TxA59JatZdzaS/nAYk7F3UxiYhIJAWEiIhEUkBUm5fsAjST9nIeoHNprdrLubSX84AEnYvGIEREJJJaECIiEkkBISIikY65gDCzPma23MzeMbP1ZvbdcHkPM3vezN4Lf3dPdlkbcphzudXMPjCzteHP/0h2WRtiZllm9rqZvRWey23h8v5m9lr4vjxuZhnJLuvhHOY85pvZ5rj3JC/ZZW0sM0s1szfN7NnweZt6T2IizqMtvydbzOwfYbkLwmXNXocdcwEBlAP/5u6nAJ8HvmVmpwI3AUvdfTCwNHze2tV3LgD3unte+PPn5BWx0Q4BZ7n7SCAPmGFmnwfuIjiXwcDHwLVJLGNj1HceADfEvSdrk1fEJvsu8E7c87b2nsTUPg9ou+8JwNSw3LHrH5q9DjvmAsLdd7j7G+HjTwn+YHoDFwILws0WABclp4SNd5hzaXM8cCB8mh7+OHAW8FS4vNW/L4c5jzbJzHKBLwAPhs+NNvaeQN3zaKeavQ475gIinpn1A0YBrwHHu/sOCCpe4Ljklazpap0LwFwz+7uZ/bYtdJdBVRfAWuAj4HlgI/CJu5eHmxTRBgKw9nm4e+w9+c/wPbnXzDKTWMSmuA+4EagMn+fQBt8T6p5HTFt8TyD40PFXM1tjZteHy5q9DjtmA8LMOgF/AP6nu+9PdnmORsS5/BoYSNDFsQP4eRKL12juXuHueUAuMBY4JWqzli1V09U+DzMbBtwMDAXGAD2Af09iERvFzM4HPnL3NfGLIzZt1e9JPecBbfA9iXOmu48GziPoWp6UiBc5JgPCzNIJKtSF7v50uHinmZ0Yrj+R4NNfqxd1Lu6+M6ykKoH/S1DZthnu/gmwgmBcpZuZpYWrcoHtySpXU8Wdx4ywO9Dd/RDwO9rGe3ImMNPMtgCPEXQt3Ufbe0/qnIeZ/XcbfU8AcPft4e+PgEUEZW/2OuyYC4iwD/Uh4B13vydu1WLgmvDxNcD/a+myNVV95xL7IwldDKxr6bI1lZn1MrNu4eMOwHSCMZXlwKxws1b/vtRzHhvi/uMaQd9wq39P3P1md891937A5cAyd7+KNvae1HMeX2qL7wmAmWWbWefYY+AcgrI3ex2W1vAm7c6ZwJeBf4T9xAD/C7gTeMLMrgW2ApcmqXxNUd+5XBF+Zc+BLcDXklO8JjkRWGBmqQQfXJ5w92fN7G3gMTO7A3iTIBBbs/rOY5mZ9SLoolkLfD2ZhTxK/07bek/qs7CNvifHA4uCXCMNeMTdnzOz1TRzHaapNkREJNIx18UkIiKNo4AQEZFICggREYmkgBARkUgKCBERiaSAEKnFzCriZvhcG05j0tRjdDOzbzZ/6URajr7mKlKLmR1w905HeYx+wLPuPqyJ+6W6e8XRvLZIc1ELQqQRwgn47jaz1eHkbl8Ll3cys6Vm9kY4P/+F4S53AgPDFsjdZjYldh+CcL//MrM54eMtZvZDM3sZuNTMBprZc+FEbC+Z2dBwu0vNbJ0F95p4sWX/BeRYdCxeSS3SkA5xV6ZvdveLCe55sM/dx4Szfv7NzP4KbAMudvf9ZtYTeNXMFhPMxT8snLQPM5vSwGuWuPuEcNulwNfd/T0zGwf8imAepB8C57r7B7HpPEQSSQEhUtdnsYo9zjnACDOLzUHUFRhMMN31j8PZNCsJpr4+/ghe83Gompn3DODJcCoFgNg01H8D5pvZE8DTdY4g0swUECKNY8C33X1JjYVBN1Ev4HR3LwtnDM2K2L+cml26tbc5GP5OIbjfQp3bX7r718MWxReAtWaW5+57juRkRBpDYxAijbME+EY4vTpmNiScSbMrwb0GysxsKvC5cPtPgc5x+78PnGpmmWbWFZgW9SLh/Tw2m9ml4euYmY0MHw9099fc/YfAbqBP85+mSDW1IEQa50GgH/BGOD30LoIpohcCz1hw4/i1wAYAd99jZn8zs3XAX9z9hrBr6O/AewSzoNbnKuDXZvYDgluWPga8BdxtZoMJWjNLw2UiCaOvuYqISCR1MYmISCQFhIiIRFJAiIhIJAWEiIhEUkCIiEgkBYSIiERSQIiISKT/Dzrdm2F5ZYq0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mf_scores = ndcg.groupby(['AlgoClass', 'features'])['ndcg'].mean().reset_index()\n",
    "pop_score = ndcg[ndcg['AlgoClass'] == 'Popular']['ndcg'].mean()\n",
    "plt.axhline(pop_score, color='grey', linestyle='--', label='Popular')\n",
    "for algo, data in mf_scores.groupby('AlgoClass'):\n",
    "    plt.plot(data['features'], data['ndcg'], label=algo)\n",
    "plt.legend()\n",
    "plt.xlabel('Features')\n",
    "plt.ylabel('nDCG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
